Have you ever did something completely different that you initially wanted to do? For me this is frequently the case. But sometimes time is not wasted and I am happy to achieve something or learn something new.
Yesterday I logged in to my laptop for half an hour to check steemit posts and ended up with evening session that took place until 2 AM local time writing SSE client with real time feed for all available fiat currencies, CFDs and derivatives from FPForex.ru. This sounds weird for me even today as yesterday I did not know that SSE exist.
I had been working before on witness ICO price feed for Golos blockchain and noticed that solution provided by 
I was curious why
I suspected that this must be some ancient AJAX code I started the debugger and whoops: inside the php page sits an iframe (another web page) generated server side using "JSP" (Java server pages). This thing contains some args after url. Digging deeeeper I found out that there are javascript modules that capture an access token from iframe url address and subscribe to SSE server to fetch data and update the tables.
Then I thought..."What the heck are Server-Sent Events (SSEs)", it is not AJAX, not websockets and found an article here https://www.html5rocks.com/en/tutorials/eventsource/basics/ with a quite good introduction to the subject.
And I came up with a crazy idea to catch this token and subscribe to a (sub)set of available instruments to get live ticker for fiat currencies, CFDs and futures directly from the server. Of course similar data are provided by brokers like IB, OANDA or other ones but 1) this was rather to solve a challenge 2) the website has a few Russian based instruments and currency pairs not easily available elsewhere.
And guess what - I succeeded. The outcome of this reverse-engineering exercise is an experimental tool https://github.com/cryptomental/social-chains-experiments/blob/master/fpticker.py that:
- Opens a keep-alive session and grabs SID token
- Subscribes to any instrument, subset, or all of them
- Streams ticker data in raw SSE format or dict/JSON
The available instruments are:
* INDEXES_CFD = ["DJIA", "SP500", "NASD_COMP", "NASD100", "FTSE100",
"DAX", "AEX" "CAC40", "SMI", "RTST", "USD_INDEX"]
* STOCKS = ["GAZ_ADR", "GMK_ADR", "LUK_ADR", "ROS_ADR", "RTK_ADR",
"SNG_ADR", "TAT_ADR", "VTB_ADR"]
* METALS = ["GOLD", "SILVER", "PLATINUM", "PALLADIUM", "ALUM",
"COPPER", "NICKEL", "BRENT", "LIGHT"]
* RUB = ["USDRUB", "EURRUB"]
* MINI = ["NASDAQ", "SnP500"],
* CURRENCY_CFD = ["AUD/JPY", "AUD/USD", "CAD/JPY", "CHF/JPY",
"EUR/AUD", "EUR/CAD", "EUR/CHF", "EUR/GBP",
"EUR/JPY", "EUR/USD", "GBP/CHF", "GBP/JPY",
"GBP/USD", "USD/CAD", "USD/CHF", "USD/JPY"]
* CURRENCY_FUTURES = ["AUD_FUT", "CAD_FUT", "CHF_FUT", "EUR_FUT",
"GBP_FUT", "JPY_FUT"]
* INDEX_FUTURES = ["DJIA_FUT", "SP500_FUT", "NASD100_FUT", "NIK_FUT"]
Raw stream example for currencies:
1;S=USDRUB;T=14:33:46;B=+64.398;NCH=0.627;NCHP=0.98;A=+64.408
1;S=EURRUB;T=14:33:57;B=-68.458;NCH=0.685;NCHP=1.01;A=-68.478
1;S=EURRUB;T=14:33:57;B=-68.457;NCH=0.684;NCHP=1.01;A=-68.477
1;S=EURRUB;T=14:33:58;B=+68.458;NCH=0.685;NCHP=1.01;A=+68.478
1;S=EURRUB;T=14:33:59;B=-68.458;NCH=0.685;NCHP=1.01;A=-68.478
Metals, stocks:
1;S=ROS_ADR;T=14:34:17;B=+5.330;NCH=0.030;NCHP=0.57;A=5.335
1;S=RTK_ADR;T=14:09:55;B=7.70;NCH=-0.11;NCHP=-1.41;A=+7.74
1;S=SNG_ADR;T=14:34:30;B=4.549;NCH=-0.041;NCHP=-0.89;A=+4.553
1;S=TAT_ADR;T=14:35:22;B=36.46;NCH=0.28;NCHP=0.77;A=-36.49
1;S=VTB_ADR;T=14:35:29;B=2.083;NCH=0.000;NCHP=0.00;A=-2.085
1;S=GOLD;T=14:35:29;B=+1211.71;NCH=-0.52;NCHP=-0.04;A=+1212.01
1;S=SILVER;T=14:27:26;B=+16.64;NCH=0.00;NCHP=0.00;A=+16.67
1;S=PLATINUM;T=13:54:22;B=+936.00;NCH=0.00;NCHP=0.00;A=+946.00
1;S=PALLADIUM;T=14:35:23;B=-743.30;NCH=1.70;NCHP=0.23;A=-746.30
1;S=ALUM;T=14:25:00;B=-1760.00;NCH=0.00;NCHP=0.00;A=-1763.00
1;S=COPPER;T=14:33:08;B=-5592.00;NCH=-20.00;NCHP=-0.36;A=-5597.00
1;S=NICKEL;T=14:18:30;B=+11302.00;NCH=4.00;NCHP=0.04;A=+11402.00
1;S=BRENT;T=14:35:16;B=-49.30;NCH=0.27;NCHP=0.55;A=-49.33
1;S=LIGHT;T=14:34:49;B=+48.20;NCH=0.30;NCHP=0.63;A=+48.23
Dict/JSON:
{'A': '+1212.44',
'B': '+1212.14',
'NCH': '-0.09',
'NCHP': '-0.01',
'S': 'GOLD',
'T': '14:42:52'}
{'A': '+16.67',
'B': '+16.64',
'NCH': '0.00',
'NCHP': '0.00',
'S': 'SILVER',
'T': '14:40:06'}
{'A': '+946.00',
'B': '+936.00',
'NCH': '0.00',
'NCHP': '0.00',
'S': 'PLATINUM',
'T': '13:54:22'}
These are just static snapshots, normally data is streamed real time. Looks pretty cool.
Treat this implementation as an experiment, or a toy, but you can access real time data with it.
This post might be too technical, but well, I am writing about some original discoveries here and actually I am writing this not to forget afterwards. I think I found another application for steemit being a diary ;)
A few references:
- Server-Sent events specification
https://html.spec.whatwg.org/multipage/comms.html#server-sent-events
- What the heck are Server-Sent Events (SSEs)