This document is intended as a open discussion with the EOS community and we encourage suggestions. Please let us know how we can better leverage the resources in Costa Rica to support EOS. We want your feedback!
The dawn of the EOS mainnet will require a great deal of coordination in order to run as smoothly as possible. Our initial priority will be providing redundant, high performance infrastructure located in Costa Rica. Once our block producing cluster is operational in Costa Rica we will deploy full nodes throughout Central America in order to provide the EOS ecosystem with low latency access in the region.
Initial Setup
We feel it is most beneficial to the network to maintain full control of all our hardware. Our proposed block producer configuration is 100% comprised of bare metal servers, located entirely within Costa Rica. This will add to the network’s resiliency by contributing technological, geographical, and political diversity as no single point of our infrastructure relies on any third party or cloud infrastructure. Everything from our .CR domain, DNS, IP addresses and servers will be 100% Costa Rican.
The current hardware characteristics for our primary block producer is:
- 4 x CPU 24 cores (48 threads) @ 1.87GHz
- 512 GB RAM
- 8 TB SSD Storage
- 100 MBps uplink
We will also launch an additional non-producing node as part of our initial setup to support our primary producer and respond in a timely manner to requests coming from users in our region.
Scaling Plan
We want to ensure that EOS scales as smoothly as possible and so improving our hardware is our main priority to handle a large number of transactions and have redundant equipment ready to go at a moments notice. We envision the following scaling plan for hardware:
Deploy two identical, primary and secondary, block producers with the following specifications:
- HP DL360 Gen 10 server
- 2 x CPU 28 cores (total 56 Cores and 112 threads) @ 2.50 GHz
- 1.5TB DDR4–2666mhz RAM Memory
- 8TB in Raid 6 configuration
- 10GB/25GB PCI fibre network card
- 1GBs uplink
Full Nodes
In addition to hosting files, block producers will be expected to run API nodes that are able to query the blockchain database state on behalf of applications. These services make it vital for applications to get the information they need without having to run and maintain their own scalable hosting services.
EOS Costa Rica will simultaneously launch two additional EOS nodes along with an IPFS storage array in Costa Rica. Once the Costa Rica server cluster is operational we will continue with the deployment of full nodes to serve the demand of Dapps in Panama, El Salvador, Guatemala, Honduras and Nicaragua, in that chronological order.
We believe block producers are incentivized to run full nodes in their regions, non-producing nodes can greatly help to reduce the load on block producing nodes by handling additional tasks that developers need such as:
Full Node Functions
- API nodes
- Seed Nodes
- Testnet nodes
- Database indexes
- Query services
- Storage
- Hosting
Costa Rica Infrastructure Diagram
Firewall
EOS Costa Rica will configure secure connections to prevent attackers from scanning the entire network topology looking for nodes to shut down. We will need to plan, communicate and work closely together with other Block Producers to ensure the network is secure and efficient.
Our block producers will not allow any incoming connections. This ensures our block producers stay hidden from the internet whilst still syncing up to the blockchain and producing blocks.
DNS
Our DNS will be hosted through one of the largest authoritative DNS networks in the world. The network has an average of a few milliseconds query speed with highly reliable service. We will run all hostnames under the Costa Rican TLD using our domain eosio.cr.
DoS
We will implement a DoS mitigation service along with Web Application Firewall (WAF), using a reverse DNS solution for all our hostnames.
Failover Protocol
We will design and implement equipment, configurations, and scripts that immediately switch block producers in case of an outage or for server maintenance.
Processes
We will define and publish official procedures for the following areas concerning a block producers responsibilities. In an effort to make the network constantly more resilient, these processes will be updated regularly as we learn from our experiences and those of other block producers. We will also will encourage feedback from the community to optimize our processes to best defend the interest of EOS token holders.
Infrastructure Availability & Application Monitoring
We monitor all systems, devices, traffic and applications 24/7. We currently use PRTG network monitor to oversee and record our systems. Sensors are configured for the following metrics.
- Response times
- Request inspection
- Transactional throughput
- TCP / HTTP checks
- Pings / Latency
Alerting / Reporting
We will document and provide visibility to our monitoring strategies. Additionally we will create a web portal for anyone to check the real time status of our infrastructure.
Regarding pageability we will establish pagerduty, escalation procedures and get an on call schedule setup. All EOS Costa Rica staff will be alerted of any outages, incidents or support requests via Email, SMS, and internal messaging service.
Incident Management
We will develop and publish a procedure describing the steps we will take to deal with any incidents and/or outages.
Changes in Capacity
We will establish processes detailing how we will deal with changes in hardware as we scale RAM, Storage, CPU
Security Monitoring
- Intrusion
- Vulnerabilities
- Traffic patterns
- Risks
Log Management Procedures
- Log Consolidation
- Debugging
- Development
- Forensics
Third Party Annual audits
- Security
- Infrastructure
- Processes
- Audit results will be shared with the EOS token holders
Future Technological Initiatives
We encourage a culture where technologists can be enthusiastic about learning and experimenting. We plan on providing an environment that fosters innovation and present the following ideas for future technical undertakings from Costa Rica.
Hosting Services
In addition to hosting files, our block producers will be running API nodes that are able to query the blockchain database state on behalf of applications. These APIs will likely be a combination of GraphQL and custom Web Assembly based queries. This is vital for applications to get the information they need without having to run and maintain their own scalable hosting services.
Testnet Nodes
We will maintain testnet nodes to use for testing internal software, procedures, and staff training. These nodes will also be available for the community to use.
High Availability Alternatives
We plan on testing and comparing different infrastructure approaches including cloud computing to benchmark performance and optimize our production infrastructure. We will test load balancers, Web Application Firewalls, DDoS mitigation strategies, and software alternatives such as docker with the goal of discovering the best way to fulfill our vision of establishing EOS high availability clusters in each Central American nation.
Custom Application Infrastructure
We are aware that there are limits to what kind of applications can be built using the general purpose infrastructure provided by block producers. Some applications may require server-side rendering or custom database indexes maintained by custom micro-services which may require custom server infrastructure hosted by the application developer or other parties. Developers of these applications can benefit from the same scalable architecture used by our block producers to deploy their own customized API and Query Services. This will help developers rapidly bring scalable application infrastructure to market.