Zomojo의 한국매매시스템 해킹하기

1.
Zomojo라고 하였지만 Zomojo를 만들었던 Matt Hurd입니다. 해킹이라고 하였지만 정확히 리버스 엔지니어링(reverse engineering) 또는 역공학(逆工學)입니다. Matt Hurd가 KSE 및 KRX의 접속환경을 분석하여 고빈도매매를 실현한 과정을 그린 자전기록을 남겼는데 The accidental HFT firm입니다.

The accidental HFT firm

Zomojo가 한국시장에서 어떤 기술을 어떻게 적용해서 얼만큼의 수익을 올렸는지는 2013년 Zomojo 판결문속에 비친 자본시장의 뒷모습을 통해 자세히 다루었습니다. 물론 판결물이 나온 배경인 분쟁에서 대해서도 Zomojo vs Zeptonics 그 이후<을 통해 다루었습니다. 이번에 쓴 글을 개발자로써 어떤 고민을 했는지를 다루고 있습니다. 한국의 트레이딩시스템 개발자와 무엇이 다른지 살펴보기 위함입니다. 2. 대략 2005년을 전후한 때로 보입니다. 매매시스템을 구축하기 위하여 리눅스서버를 도입했다는 문장이 들어옵니다. 64비트와 32비트, 성능과 안정성을 두고 고민을 하는 점이 인상적입니다. 이 때 Sun사의 Solaris가 한국 트레이딩시스템시장의 다수였고 IBM의 AIX가 세력을 넓혀가던 때입니다.

So with a 1.5 million AUD investment, it was time to start working again, all by myself with a laptop in a serviced office. I built some servers, desktops, installed Suse 64-bit Linux, hired three programmers – and the funding runway clock started ticking. It turned out I’m not a very good Linux system administrator. Not many months later I tired of doing system-administration badly and got some part-time help. Our head-count was now four and a half. But 64-bit Linux, especially with AMD & Hypertransport, was not ready for prime time, so we dialled back to 32-bit and Redhat / CentOS. It is a wise call from Mr. Sysadmin and progress improved.

양자선택의 기로에서 한국시장을 선택하고 진출할 때가 KRX를 위한 통합논의가 한참 진행중이던 때입니다. 이 때나 지금이나 자본시장법(증권거래법)에 의해 투자자는 반드시 증권사(선물사)의 매매시스템을 통하여 거래소 시스템에 접속할 수 있습니다. 외국인투자자도 마찬가지입니다. 보통 FEP라고 하지만 아래글에서는 Gateway라고 지칭합니다.

In those old times in Korea, traders, even HFT ones, didn’t get access to the exchange directly. You had to go through a broker gateway and use their broker specific API to connect. There was no co-location. A great deal of the variation in performance, beyond a broker’s control, was simply due to the location and exchange-provided technology. Now, the KOFFEX was the futures and option trading bit. KOFFEX was based in Busan, a seaport town a little over 300km away from Seoul. The KOSPI 200 products were small in ticks but large in volume. They were the most popular derivative contracts in the world by around a factor of nearly ten back then. Eurodollars at CME were number two and K200 options were number one in volume. This had proved a bit too much to handle for KOFFEX. They had the K200 market run, on their behalf, by KSE in Seoul. So everything we needed was in one location in Seoul, although the primary derivative platform was based in Busan.

Now, I was aware that the network stack was perhaps the most important latency aspect I could control in this implementation. I found a hardware device that could do a three-microsecond translation from InfiniBand to Ethernet IP. It was a module that would plug into a TopSpin InfiniBand switch. Hypertransport (HTX), an alternative to PCI Express (PCIe), was a new thing in AMD land and it provided a lower latency way of doing InfiniBand comms. I bought some Pathscale HTX Infinipath cards, HTX mainboards in alpha (e.g. serial number 0x0000045!), boxes, plus assorted bits. I assembled up a motley crew of parts to be a gee-whiz, not dirt cheap, but extremely frugal, network stack. It felt good. I had my somewhat clumsy but world-competitive network stack. It would not be for some years that regular network cards could spank that set-up.

글중에 나오는 Pathscale HTX Infinipath card가 시장에 나온 때가 2005년입니다. 무척 앞선 노력이었습니다. 국내투자자도 속도에 민감하였지만 대부분 HTS를 통한 주문을 좀더 빨리하는 방법에 집중할 뿐이었습니다.

PathScale Releases InfiniPath 1.1 Software

글을 읽다가 보면 역공학이 해외에서 일반적임을 알 수 있습니다. 더 빠른 속도를 위함임을 물론입니다.

The new Sangoma cards supported a transparent bitstream approach where it just dumped the raw bits out on a channel you could slurp. I started hacking away and figured out what the streams looked like. X.25 virtual channels were grouped into data feeds for market data. They tended to be consecutive. There was a different channel group for puts, calls, and futures. I had to hack up a brumby HDLC layer but before long I had packets corresponding to the KRX feed. Bypassing the modem and network device saved about a millisecond.

This wasn’t unfamiliar territory. Eurex, when it had the old Values API, provided by a box, people hacked around it and reverse engineered the protocol. A formal Technical Member Regulation document was adjusted to make that against the exchange rules. The practice was stopped. There was no such rule in Korea. Similarly, when the ASX opened its first small co-location in Bondi Junction, I noted a couple of firms had taps around the exchange provided 1U Dell server gateways. Reverse engineering the Nasdaq ring protocol wasn’t as uncommon as I had imagined; there are lots of clever hackers out there.

KRX로 전환한 이후 가장 집중했던 분야는 시장정보 속도입니다. 각 증권사마다 속도에서 차이가 있어서 이를 측정하기 위해 CDMA Timer를 적용했다고 합니다.

We did the same kind of set up elsewhere and kept on going. We noticed the market data speeds were quite different at different sites. We couldn’t get GPS into the sites but Korea used a CDMA mobile phone system. Endrun Technologies had a nice little plastic box with which you could upgrade the temperature-controlled oscillator and get stable timing. This device syphoned off the embedded GPS signal within the CDMA signal and provided an NMEA serial feed plus a pulse per second (PPS) cable. CDMA is only spec’d to 10 microseconds of jitter, but we were getting around one microsecond of accuracy in Seoul. FWIW, Canada’s CDMA gave us around 5 or 6 microseconds of accuracy.

We did eventually end up with systems in the KRX underground data centre itself. It was quite the security rigmarole to get in. It was a long way underground but I noticed I could get a cell phone signal in the DC. I put a CDMA timer in there too. It worked enough that we got good timing from the intermittent cell signal until something burnt out. I suspected it needed a better antenna as perhaps something was driving too hard. Before a trip to Seoul, I bought a meter long white whip antenna and soldered on an SMA connector. I got into the KRX DC underground bunker and laid the big white antenna on top of our rack. The Koscom guys looked at it suspiciously. I don’t know what they thought, but a surprising number of other people over the years mentioned they had seen my white whip aerial laying on top a rack in the KRX DC.

We now had good timing at the sites and needed to share data. We found that within a couple of blocks of the exchange campus, we were getting 300-microsecond one-way traffic over 10Mb public Internet Ethernet between our sites. That was surprisingly good. We co-ordinated our market data and ordering over those internet lines. Korea has great internet and it was certainly good enough back then. Milliseconds of advantage from consistent market data across all sites improved trading. Whilst our best ordering came from the exchange campus, the best market data came from a couple of blocks away. That was unexpected. It remained the best market data site for a few years.

이상은 X.25가 표준이었던 시대입니다. TCP/IP가 표준으로 등장한 Exture부터 새로운 도전을 합니다. Exture이후 핵심적인 과제는 FEP간의 암호화였습니다. X.25를 대체한 TCP/IP네트워크의 보안을 위해 FEP사이에 하드웨어를 포함한 암호화를 적용합니다. 이를 해킹하는 것입니다. ARIA 소스를 확보하고 Bypass하는 기술 및 packet fragmentation technique을 연구하였습니다.

I wrote to the university in Seoul that was the keeper of ARIA, and they sent me some 8 and 32-bit C code for the cipher. We took that, broke into enough of the handshake to get the session key, and we could then encipher messages correctly. There were some dodgy aspects of the whole encryption business model as you had to pay a firm a royalty of a few thousand dollars a year to use their official, open-source derived, encryption library. Everyone in Korea was paying for this. Someone was making out like a bandit.

By using our own ARIA bypass, we were a lot faster without the official cipher library. Eventually, we wrote own custom SIMD version that worked even better. Our reverse-engineered encryption allowed our whole trade cycle overhead to be less than just the enforced official encryption overhead.
Now we could finally send a partial order message down the TCP stream. We saved some milliseconds doing this which got eroded to hundreds of microseconds over time as the line speeds increased. The benefit was actually greater than just the stream latency savings. The switches Koscom used were “store and forward switches” back then, so smaller packets also transitioned faster through the whole system. The improvements in timings ended up being a bit better than we expected perhaps due to this.

The hack was to force those cypher zeros into the user field and to also keep the middle fragment with a few characters of that same user field. The packets split the user field. You have a better than ninety percent chance the junk you have in the tail user field is non-zero on your first crack. If you check this when preparing your middle packet and you hit a problem, you just cycle through a character or two in the small held out user field in the middle packet, that is the final packet delivered until you’re zero free in the deciphered interpretation of the tail already sent. This hack can save you a large amount of latency.

3.
이상을 읽으면서 해외개발자들이 레이턴시를 줄이기 위해 투입하는 노력이 상상이었습니다. 그런데 의문이 들었습니다. 국내트레이더들이 이런 노력을 하려고 할 때 할 기회가 있었을까요? 여기서 투자자가 공급한 FEP가 문제로 등장합니다. 국내투자자가 고객사의 FEP를 대신하여 자체적으로 독자개발한 FEP를 이용하여 주문시스템을 구축하는 것이 가능할까요? 불가능합니다만 외국인투자자에겐 가능했습니다. 이를 아래와 같이 표현합니다. 불법은 아니라고 하더라도 탈법임은 명확합니다.

This broker was a bit smarter and understood that if we were tapping lines then we could potentially see orders flowing the other way. They were right, we could, but weren’t doing that yet. We had to get a separate infrastructure for our own market data set-up. The broker did a great job and we were thankful for the smooth transition. The Muppetz Experience proved to be an exception. Brokers in Korea are good people.

그리고 서울과 부산사이에 시세속도를 높이기 위해 다양한 방식이 적용되었고 이를 묵인했음도 확인할 수 있습니다. 서울과 부산 IT센터의 전용회선 속도를 높이기 위해 코스콤으로부터 별도의 회선을 임대한 사실, KT로부터 마이크로웨이브서비스를 이용한 점도 확인할 수 있습니다.

A lot of money had been spent on data centres by banks and brokers in Seoul. To mitigate this, market data was sent to Seoul for use and then bounced back to Busan for use in the proximity centre there. This was a mess. Only now, in 2016, has the KRX moved to have local data in Busan. Prior to having local data in 2016, the best solution was to put your market data in Seoul and pipe data down to ordering infrastructure in Busan. Costly and inefficient. Various firms, like us, considered getting their own lines between Seoul and Busan. Koscom banned that to keep the revenue. You could only buy – various expensive – lines from Koscom, so brokers did. Generally, you were fighting over lines that were 2.9 or 3.1 milliseconds in speed for the link. Quite slow for a 300km direct, or a windy 400km drive. There have been a couple of KT-based part microwave solutions pop up though in the last couple of years.

마지막으로 Exture이든 Exture+이든 트레이더에게 중요한 것은 Latency가 아니라 Jitter임을 확인했습니다.

At the same time, the KRX moved from being the largest commercial HP/Compaq/Tandem platform in the world to an AIX platform running on IBM PowerPC architecture. Before the move, we were getting around 11-12 ms round trip times (RTT) on the order lines with a jitter of about 130 ms. After the Busan move, RTTs blew out to over 20ms but the jitter dropped to around 30ms. Even though the latency doubled, the jitter reduction made being fast more productive.

무척이나 공들인 글입니다. 자신이 시험했던 기술들은 소상히 기록했습니다. 어떤 의미가 있든 알고리즘트레이딩을 하시는 분이면 한번쯤 읽어보셨으면 합니다.