As you know a full bitcoin node requires a monster machine as the blockchain is almost 75GB. I was battling against startup time to get to realtime sync for bitcoin and on the non-SSD server it was taking forever.
well not forever, but a big fraction of the time it took to sync all other block. which is clearly silly, but I did really optimize the parallel sync so any unoptimized syncing will be rather slow compared to that. I tried the usual tricks of lump allocation of memory, prefetching the memory mapped files and it made incremental improvements but I just couldnt stand waiting so long for it to sync.
So I had an idea.
Each coin has a bundle of blocks and the bundlesize I set to be the 500 or 2000 that comes back from the headers message as that is the "natural" size for a bundle. However, while it is a bit unnatural to use a larger bundle size since you probably wont get anymore than 2000 items in the reply, nothing prevents setting an arbitrarily small bundlesize. Unless the software has some assumption somewhere about being 500 or 2000, I should be able to just change one line of code and make bitcoin have a smaller bundlesize.
I set it to 100, which is 20x smaller and it went from requiring a 32GB RAM system with SSD to where it is doing the parallel sync now using a bit more than 2GB of RAM. even the tmp files used is peaking at 15GB instead of 30GB like before. With much less stress on the system limits things should operate much more linearly and I am seeing it syncing a lot smoother on the slower VPS. It usually took a bit more than 2hrs for sync, now it is 90%+ done after 1.5 hours. It seems to not be using more than 200mbps of bandwidth, so the <2hr time would be possible if you have a 200mbps connection, 4hrs at 100mbps, 6 to 8hrs at 20mbps. I know the last one doesnt add up, but it is due to the ability of iguana to totally saturate a connection, so it will sustain a bit over the rated capacity of the line the entire duration of the sync. while at faster speeds it goes up and down in usage
I estimate the time to achieve RT mode will be 20x faster and so on average you will be fully in sync before the first bitcoin block comes in, which seems to be adequate.
This changes the expected configuration from being basilisk BTC to iguana BTC and will allow for better performance overall. HDD space seems to be a lot more plentiful than RAM and I think most systems can handle 2GB of RAM used for bitcoin. This is a full bitcoin node with block explorer level dataset and will allow full functionality without reliance on any other nodes.
In some ways this is one of the most significant iguana news in a while
took a bunch of restarts after crashes to get it to stumble to the finish line, but all in all very good result to sync using much less RAM. Since there is no iguana database to corrupt, you can almost always just restart after any crash, at anytime. iguana tries to identify what is needed to be done and just resumes, this part is a bit ad hoc and it doesnt always figure out the most efficient things to do, but by deleting the right intermediate files iguana is most always regenerating what is needed. I guess worst case is to do a full sync from scratch and burn 2 hours.
Tables creation without SSD takes a while, almost 2 hours with the increased number of bundles and RAM usage peaked at 10GB, so i will need to do some optimizations to fit it into 4GB, though this was using 8 cores, so if to use only 2 cores it would still fit in 4GB. But being able to run BTC along with a dozen other coins is worth the slower table generation time, especially since most connections will be taking longer than 1hr for the downloading part anyway
Quite an unexpected development today! I was going to debug basilisk, but ended up reducing BTC requirements to be able to run on 4GB systems. Of course more RAM will be better performance and SSD is highly recommended, but always good to have the lowest hardware requirements.
James
The big question is if the two bundlesizes created the same ledger, I know you were wondering that, so here is the data:
2000 block bundles:
BTC.RT426740 u.213+c.213 b.213 v.213 (0+740/740 1st.213).s0 to 213 N[214] h.426740 r.426740 c.426000 s.426740 d.0 E.213 maxB.16 peers.118/64 Q.(0 0) (L.426740 213:740) M.426739 00000000000000000207f8109a1c8b5860950377f5d1e835da95c190012e34d6 ledger.cab1009145174be1 supply 15834239.79530316 bQ.3 8:33:09 stuck.0 max.0
100 block bundles:
BTC.RT426740 u.4267+c.4267 b.4267 v.4267 (0+40/40 1st.4267).s0 to 4267 N[4268] h.426740 r.426740 c.426700 s.426740 d.0 E.4267 maxB.16 peers.117/64 Q.(0 0) (L.426740 4267:40) M.426739 00000000000000000207f8109a1c8b5860950377f5d1e835da95c190012e34d6 ledger.7f53a8f59cd4974f supply 15834239.79530316 bQ.2 0:41:47 stuck.0 max.0
notice the supply for both: 15834239.79530316 vs 15834239.79530316
matching supply doesnt mean 100% that all the ledger entries are identical, but it is a very very good chance, especially since all the bundles were verified in isolation and all the balances cross checked against listunspent output