Request for Expert Assistance: Receiving Real-Time Stock Data in JMP Using JSL and Python

[#1]

I’m working on a project to receive real-time stock market data using JMP software and need your guidance. JMP supports its scripting language (JSL) and embedded Python execution, which I’d like to leverage. My goal is to capture specific stock data fields from a 32-bit DLL (StockDrv.dll) and store them in a JMP data table. I’ve developed a solution, but I’d greatly appreciate your review and suggestions to ensure it’s robust and efficient.

Background

I have a stock data client installed at I:\DZH\HailiLive\HailiLive.EXE, which provides a 32-bit DLL (I:\DZH\HailiLive\StockDrv.dll). This DLL delivers real-time stock data via Windows messages. The data is structured in a format called RCV_REPORT_STRUCTExV3, sent through a custom message (WM_APP + 1) with a work mode of RCV_WORK_SENDMSG (value 4). The relevant header file (Stockdrv.h) defines the DLL’s interface, including key functions like Stock_Init and Stock_Quit, and the data structure.

I need only the following fields:

Market: Stock market identifier (e.g., 'HS' for Shanghai).
Code: Stock code (e.g., "600050").
Name: Stock name.
Time: Transaction time.
LastClose: Previous closing price.
Open: Opening price.
High: Highest price.
Low: Lowest price.
NewPrice: Latest price.
Volume: Transaction volume.
Amount: Transaction amount.
BuyPrice1 to BuyPrice5: Five levels of bid prices.
BuyVolume1 to BuyVolume5: Five levels of bid volumes.
SellPrice1 to SellPrice5: Five levels of ask prices.
SellVolume1 to SellVolume5: Five levels of ask volumes.

[#2]

The data is wrapped in a RCV_DATA structure, where m_pReportV3 points to an array of RCV_REPORT_STRUCTExV3 records. JSL can load the DLL and parse binary data using Peek, while Python can handle the Windows message loop via ctypes.

Challenge

JSL alone cannot create a Windows message loop, so I’ve integrated Python to manage this. The DLL is initialized with a window handle and message ID, then sends data to that window. I need to parse this data in JSL and populate a JMP table with the specified fields.

My Proposed Solution

Below is my implementation. It uses JSL to define the table and parse data, and Python to handle the message loop. I’ve included comments to explain each step. Could you please review it and suggest improvements?

// StockReceiver.jsl

// Create a JMP data table with required columns
dt = New Table("Stock Data",
    New Column("Market", Character, Width(4)),        // Market identifier (e.g., 'HS')
    New Column("Code", Character, Width(10)),         // Stock code
    New Column("Name", Character, Width(32)),         // Stock name
    New Column("Time", Numeric, Format("yyyy-mm-dd hh:mm:ss")), // Transaction time
    New Column("LastClose", Numeric, Format("Fixed Dec", 12, 2)), // Previous close
    New Column("Open", Numeric, Format("Fixed Dec", 12, 2)),      // Opening price
    New Column("High", Numeric, Format("Fixed Dec", 12, 2)),      // Highest price
    New Column("Low", Numeric, Format("Fixed Dec", 12, 2)),       // Lowest price
    New Column("NewPrice", Numeric, Format("Fixed Dec", 12, 2)),  // Latest price
    New Column("Volume", Numeric, Format("Fixed Dec", 12, 2)),    // Volume
    New Column("Amount", Numeric, Format("Fixed Dec", 12, 2)),    // Amount
    New Column("BuyPrice1", Numeric, Format("Fixed Dec", 12, 3)), // Bid price 1
    New Column("BuyVolume1", Numeric, Format("Fixed Dec", 12, 2)),// Bid volume 1
    New Column("SellPrice1", Numeric, Format("Fixed Dec", 12, 3)),// Ask price 1
    New Column("SellVolume1", Numeric, Format("Fixed Dec", 12, 2)),// Ask volume 1
    New Column("BuyPrice2", Numeric, Format("Fixed Dec", 12, 3)), // Bid price 2
    New Column("BuyVolume2", Numeric, Format("Fixed Dec", 12, 2)),// Bid volume 2
    New Column("SellPrice2", Numeric, Format("Fixed Dec", 12, 3)),// Ask price 2
    New Column("SellVolume2", Numeric, Format("Fixed Dec", 12, 2)),// Ask volume 2
    New Column("BuyPrice3", Numeric, Format("Fixed Dec", 12, 3)), // Bid price 3
    New Column("BuyVolume3", Numeric, Format("Fixed Dec", 12, 2)),// Bid volume 3
    New Column("SellPrice3", Numeric, Format("Fixed Dec", 12, 3)),// Ask price 3
    New Column("SellVolume3", Numeric, Format("Fixed Dec", 12, 2)),// Ask volume 3
    New Column("BuyPrice4", Numeric, Format("Fixed Dec", 12, 3)), // Bid price 4
    New Column("BuyVolume4", Numeric, Format("Fixed Dec", 12, 2)),// Bid volume 4
    New Column("SellPrice4", Numeric, Format("Fixed Dec", 12, 3)),// Ask price 4
    New Column("SellVolume4", Numeric, Format("Fixed Dec", 12, 2)),// Ask volume 4
    New Column("BuyPrice5", Numeric, Format("Fixed Dec", 12, 3)), // Bid price 5
    New Column("BuyVolume5", Numeric, Format("Fixed Dec", 12, 2)),// Bid volume 5
    New Column("SellPrice5", Numeric, Format("Fixed Dec", 12, 3)),// Ask price 5
    New Column("SellVolume5", Numeric, Format("Fixed Dec", 12, 2)) // Ask volume 5
);

// Global variables for managing the receiver
global:hwnd = 0;     // Window handle
global:dll = 0;      // DLL object
global:running = 1;  // Flag to control the message loop

// Function to parse RCV_DATA and extract RCV_REPORT_STRUCTExV3 fields
ParseStockData = Function({lParam},
    // Check the data type (RCV_REPORT = 0x3f001234)
    dataType = Peek(lParam, 0, "int");
    If(dataType == 1061153620, // RCV_REPORT
        // Get number of records and pointer to report data
        packetNum = Peek(lParam, 4, "int");          // m_nPacketNum
        reportPtr = Peek(lParam, 276, "int");        // m_pReportV3 offset
        
        // Process each record
        For(i = 0, i < packetNum, i++,
            baseOffset = reportPtr + i * Peek(reportPtr, 0, "short"); // Offset based on m_cbSize
            
            // Extract fields from RCV_REPORT_STRUCTExV3
            time = Peek(baseOffset, 2, "int");       // m_time (time_t)
            market = Peek(baseOffset, 6, "short");   // m_wMarket
            code = Peek(baseOffset, 8, "char[10]");  // m_szLabel
            name = Peek(baseOffset, 18, "char[32]"); // m_szName
            lastClose = Peek(baseOffset, 50, "float"); // m_fLastClose
            open = Peek(baseOffset, 54, "float");    // m_fOpen
            high = Peek(baseOffset, 58, "float");    // m_fHigh
            low = Peek(baseOffset, 62, "float");     // m_fLow
            newPrice = Peek(baseOffset, 66, "float"); // m_fNewPrice
            volume = Peek(baseOffset, 70, "float");  // m_fVolume
            amount = Peek(baseOffset, 74, "float");  // m_fAmount
            buyPrice1 = Peek(baseOffset, 78, "float");  // m_fBuyPrice[0]
            buyPrice2 = Peek(baseOffset, 82, "float");  // m_fBuyPrice[1]
            buyPrice3 = Peek(baseOffset, 86, "float");  // m_fBuyPrice[2]
            buyPrice4 = Peek(baseOffset, 126, "float"); // m_fBuyPrice4
            buyPrice5 = Peek(baseOffset, 134, "float"); // m_fBuyPrice5
            buyVolume1 = Peek(baseOffset, 90, "float");  // m_fBuyVolume[0]
            buyVolume2 = Peek(baseOffset, 94, "float");  // m_fBuyVolume[1]
            buyVolume3 = Peek(baseOffset, 98, "float");  // m_fBuyVolume[2]
            buyVolume4 = Peek(baseOffset, 130, "float"); // m_fBuyVolume4
            buyVolume5 = Peek(baseOffset, 138, "float"); // m_fBuyVolume5
            sellPrice1 = Peek(baseOffset, 102, "float"); // m_fSellPrice[0]
            sellPrice2 = Peek(baseOffset, 106, "float"); // m_fSellPrice[1]
            sellPrice3 = Peek(baseOffset, 110, "float"); // m_fSellPrice[2]
            sellPrice4 = Peek(baseOffset, 142, "float"); // m_fSellPrice4
            sellPrice5 = Peek(baseOffset, 150, "float"); // m_fSellPrice5
            sellVolume1 = Peek(baseOffset, 114, "float"); // m_fSellVolume[0]
            sellVolume2 = Peek(baseOffset, 118, "float"); // m_fSellVolume[1]
            sellVolume3 = Peek(baseOffset, 122, "float"); // m_fSellVolume[2]
            sellVolume4 = Peek(baseOffset, 146, "float"); // m_fSellVolume4
            sellVolume5 = Peek(baseOffset, 154, "float"); // m_fSellVolume5
            
            // Convert time_t to readable format (milliseconds for JMP)
            timeStr = Format(DateTime(time * 1000), "yyyy-mm-dd hh:mm:ss");
            
            // Add row to table
            dt << Add Rows(1);
            row = N Rows(dt);
            :Market[row] = Char(market);     // Convert market code to string
            :Code[row] = code;
            :Name[row] = name;
            :Time[row] = timeStr;
            :LastClose[row] = lastClose;
            :Open[row] = open;
            :High[row] = high;
            :Low[row] = low;
            :NewPrice[row] = newPrice;
            :Volume[row] = volume;
            :Amount[row] = amount;
            :BuyPrice1[row] = buyPrice1;
            :BuyPrice2[row] = buyPrice2;
            :BuyPrice3[row] = buyPrice3;
            :BuyPrice4[row] = buyPrice4;
            :BuyPrice5[row] = buyPrice5;
            :BuyVolume1[row] = buyVolume1;
            :BuyVolume2[row] = buyVolume2;
            :BuyVolume3[row] = buyVolume3;
            :BuyVolume4[row] = buyVolume4;
            :BuyVolume5[row] = buyVolume5;
            :SellPrice1[row] = sellPrice1;
            :SellPrice2[row] = sellPrice2;
            :SellPrice3[row] = sellPrice3;
            :SellPrice4[row] = sellPrice4;
            :SellPrice5[row] = sellPrice5;
            :SellVolume1[row] = sellVolume1;
            :SellVolume2[row] = sellVolume2;
            :SellVolume3[row] = sellVolume3;
            :SellVolume4[row] = sellVolume4;
            :SellVolume5[row] = sellVolume5;
        );
        Return(1); // Message processed
    ,
        Return(0); // Unknown data type, not processed
    );
);

// Load the DLL
dll = Load DLL("I:\DZH\HailiLive\StockDrv.dll");
If(Is Empty(dll),
    Throw("Failed to load StockDrv.dll");
);
global:dll = dll;

// Run Python code to handle Windows message loop
Run Python(
    "
    import ctypes
    from ctypes import wintypes
    import jmp

    # Constants for message handling
    WM_APP = 0x8000
    My_Msg_StkData = WM_APP + 1
    RCV_REPORT = 0x3f001234

    # Windows API type definitions
    HWND = wintypes.HWND
    UINT = wintypes.UINT
    WPARAM = wintypes.WPARAM
    LPARAM = wintypes.LPARAM
    WNDPROC = ctypes.WINFUNCTYPE(ctypes.c_long, HWND, UINT, WPARAM, LPARAM)

    # Load user32.dll for Windows API calls
    user32 = ctypes.WinDLL('user32')

    # Define WNDCLASSEX structure for window registration
    class WNDCLASSEX(ctypes.Structure):
        _fields_ = [
            ('cbSize', UINT),
            ('style', UINT),
            ('lpfnWndProc', WNDPROC),
            ('cbClsExtra', ctypes.c_int),
            ('cbWndExtra', ctypes.c_int),
            ('hInstance', wintypes.HINSTANCE),
            ('hIcon', wintypes.HICON),
            ('hCursor', wintypes.HCURSOR),
            ('hbrBackground', wintypes.HBRUSH),
            ('lpszMenuName', wintypes.LPCWSTR),
            ('lpszClassName', wintypes.LPCWSTR),
            ('hIconSm', wintypes.HICON)
        ]

    # Window procedure to handle messages
    @WNDPROC
    def WndProc(hWnd, msg, wParam, lParam):
        if msg == My_Msg_StkData and wParam == RCV_REPORT:
            # Call JSL to process the data
            result = jmp.run_jsl('ParseStockData(' + str(lParam) + ');')
            return result
        return user32.DefWindowProcW(hWnd, msg, wParam, lParam)

    # Get global variables from JSL
    hInstance = ctypes.windll.kernel32.GetModuleHandleW(None)  # Current module handle
    dll = jmp.get_global('dll')                                # DLL object
    running = jmp.get_global('running')                        # Loop control flag

    # Register window class
    wcex = WNDCLASSEX()
    wcex.cbSize = ctypes.sizeof(WNDCLASSEX)
    wcex.style = 0x3  # CS_HREDRAW | CS_VREDRAW
    wcex.lpfnWndProc = WNDPROC(WndProc)
    wcex.cbClsExtra = 0
    wcex.cbWndExtra = 0
    wcex.hInstance = hInstance
    wcex.hIcon = 0
    wcex.hCursor = 0
    wcex.hbrBackground = 0
    wcex.lpszMenuName = None
    wcex.lpszClassName = 'StockDataWindow'
    wcex.hIconSm = 0
    if not user32.RegisterClassExW(ctypes.byref(wcex)):
        raise Exception('Failed to register window class')

    # Create window
    hWnd = user32.CreateWindowExW(
        0, 'StockDataWindow', 'Stock Receiver', 0,
        0, 0, 100, 100, 0, 0, hInstance, None
    )
    if not hWnd:
        raise Exception('Failed to create window')
    jmp.set_global('hwnd', hWnd)  # Store window handle in JSL

    # Initialize the DLL
    stock_init = dll.Stock_Init
    stock_init.argtypes = [HWND, UINT, ctypes.c_int]
    stock_init.restype = ctypes.c_int
    result = stock_init(hWnd, My_Msg_StkData, 4)  # 4 = RCV_WORK_SENDMSG
    if result != 1:
        raise Exception('Stock_Init failed: ' + str(result))

    # Message loop
    msg = ctypes.wintypes.MSG()
    while running[0]:
        if user32.GetMessageW(ctypes.byref(msg), 0, 0, 0) > 0:
            user32.TranslateMessage(ctypes.byref(msg))
            user32.DispatchMessageW(ctypes.byref(msg))
        else:
            break

    # Cleanup
    stock_quit = dll.Stock_Quit
    stock_quit.argtypes = [HWND]
    stock_quit.restype = ctypes.c_int
    stock_quit(hWnd)
    user32.DestroyWindow(hWnd)
    user32.UnregisterClassW('StockDataWindow', hInstance)
    print('Stock data receiver stopped')
    ",
    "Stock data receiver started"  // Output when Python starts
);

// Function to stop the receiver
StopReceiver = Function({},
    global:running = 0;  // Signal Python to exit loop
    If(global:hwnd != 0,
        Call DLL("Stock_Quit", dll, global:hwnd);  // Cleanup DLL
        global:hwnd = 0;
    );
    Write("Stock data receiver stopped.\!n");
);

// Display startup message
Write("Stock data receiver initialized. Run StopReceiver() to stop.\!n");

Key Details

• DLL Functions:
  • Stock_Init(HWND hWnd, UINT Msg, int nWorkMode): Initializes the DLL, returns 1 on success.
  • Stock_Quit(HWND hWnd): Stops message sending, returns 1 on success.
• Data Structure Offsets:
  • RCV_DATA: m_pReportV3 at byte 276 (after m_wDataType, m_nPacketNum, m_File, m_bDISK).
  • RCV_REPORT_STRUCTExV3: Fields aligned with #pragma pack(1), e.g., m_time at 2, m_fBuyPrice5 at 134, m_fSellVolume5 at 154.
• Execution:
  • JSL loads the DLL and defines the table.
  • Python creates a window, initializes the DLL, and runs the message loop.
  • Messages trigger ParseStockData to populate the table.

Questions for You

1. Is the binary parsing using Peek correct for the offsets? I assumed no padding due to #pragma pack(1).
2. Could the message loop be optimized for high-frequency data?
3. Should I add buffering to reduce table updates (e.g., every 100 rows)?

I’d be grateful for your feedback on potential issues, performance enhancements, or alternative approaches. Thank you in advance for your expertise!

Best regards,Thanks Experts!

[#3]

Of course, I let grok3 synthesize these contents according to two C++ codes and related interface specifications.

Whether this is feasible.I sincerely need the help of experts.

[#4]

Seems like a complicated question.
Finally, the grok3 feed scheme was adopted:
Use the original C code with MinGW Installer to create a 32-bit EXE to listen to the 32-bit DLL to receive data.
This eliminates the need to install C++Builder XE7.
Data is still processed using 64-bit JMP.

Thanks Experts!