======== conn.log ======== The connection log, or :file:`conn.log`, is one of the most important logs Zeek creates. It may seem like the idea of a “connection” is most closely associated with stateful protocols like Transmission Control Protocol (TCP), unlike stateless protocols like User Datagram Protocol (UDP). Zeek’s :file:`conn.log`, however, tracks both sorts of protocols. This section of the manual will explain key elements of the :file:`conn.log`. The Zeek script reference, derived from the Zeek code, completely explains the meaning of each field in the :file:`conn.log` (and other logs). It would be duplicative to manually recreate that information in another format here. Therefore, this entry seeks to show how an analyst would make use of the information in the conn.log. Those interested in getting details on every element of the :file:`conn.log` should reference :zeek:see:`Conn::Info`. For additional explanation, including Zeek's notions of originator and responder, see :ref:`writing-scripts-connection-record`. Throughout the sections that follow, we will inspect Zeek logs in JSON format. Inspecting the :file:`conn.log` =============================== To inspect the :file:`conn.log`, we will use the same techniques we learned in the last section of the manual. First, we have a JSON-formatted log file, either collected by Zeek watching a live interface, or by Zeek processing stored traffic. We use the :program:`jq` utility to review the contents. .. code-block:: console zeek@zeek:~zeek-test/json$ jq . -c conn.log :: {"ts":1591367999.305988,"uid":"CMdzit1AMNsmfAIiQc","id.orig_h":"192.168.4.76","id.orig_p":36844,"id.resp_h":"192.168.4.1","id.resp_p":53,"proto":"udp","service":"dns","duration":0.06685185432434082,"orig_bytes":62,"resp_bytes":141,"conn_state":"SF","missed_bytes":0,"history":"Dd","orig_pkts":2,"orig_ip_bytes":118,"resp_pkts":2,"resp_ip_bytes":197,"ip_proto":17} {"ts":1591367999.430166,"uid":"C5bLoe2Mvxqhawzqqd","id.orig_h":"192.168.4.76","id.orig_p":46378,"id.resp_h":"31.3.245.133","id.resp_p":80,"proto":"tcp","service":"http","duration":0.25411510467529297,"orig_bytes":77,"resp_bytes":295,"conn_state":"SF","missed_bytes":0,"history":"ShADadFf","orig_pkts":6,"orig_ip_bytes":397,"resp_pkts":4,"resp_ip_bytes":511,"ip_proto":6} Alternatively, we could see each field printed on its own line: .. code-block:: console zeek@zeek:~zeek-test/json$ jq . conn.log :: { "ts": 1591367999.305988, "uid": "CMdzit1AMNsmfAIiQc", "id.orig_h": "192.168.4.76", "id.orig_p": 36844, "id.resp_h": "192.168.4.1", "id.resp_p": 53, "proto": "udp", "service": "dns", "duration": 0.06685185432434082, "orig_bytes": 62, "resp_bytes": 141, "conn_state": "SF", "missed_bytes": 0, "history": "Dd", "orig_pkts": 2, "orig_ip_bytes": 118, "resp_pkts": 2, "resp_ip_bytes": 197, "ip_proto": 17 } { "ts": 1591367999.430166, "uid": "C5bLoe2Mvxqhawzqqd", "id.orig_h": "192.168.4.76", "id.orig_p": 46378, "id.resp_h": "31.3.245.133", "id.resp_p": 80, "proto": "tcp", "service": "http", "duration": 0.25411510467529297, "orig_bytes": 77, "resp_bytes": 295, "conn_state": "SF", "missed_bytes": 0, "history": "ShADadFf", "orig_pkts": 6, "orig_ip_bytes": 397, "resp_pkts": 4, "resp_ip_bytes": 511, "ip_proto": 6 } What an analyst derives from any log is a function of the questions that he or she is trying to ask of it. The :file:`conn.log` primarily captures so-called “layer 3” and “layer 4” elements of network activity. This is essentially who is talking to whom, when, for how long, and with what protocol. Understanding the Second :file:`conn.log` Entry =============================================== Let’s use this framework to parse the two log entries. We will start with the second entry first. I will explain why shortly. For reference, that entry is the following: :: { "ts": 1591367999.430166, "uid": "C5bLoe2Mvxqhawzqqd", "id.orig_h": "192.168.4.76", "id.orig_p": 46378, "id.resp_h": "31.3.245.133", "id.resp_p": 80, "proto": "tcp", "service": "http", "duration": 0.25411510467529297, "orig_bytes": 77, "resp_bytes": 295, "conn_state": "SF", "missed_bytes": 0, "history": "ShADadFf", "orig_pkts": 6, "orig_ip_bytes": 397, "resp_pkts": 4, "resp_ip_bytes": 511, "ip_proto": 6 } For the second log, ``192.168.4.76`` talked to ``31.3.245.133``. The log *timestamp*, indicated by the ``ts`` field, is ``1591367999.430166``, which translates as shown below, courtesy of the Unix :program:`date` command: .. code-block:: console zeek@zeek:~zeek-test/json$ date -d @"1591367999.430166" :: Fri Jun 5 14:39:59 UTC 2020 The two systems conversation only lasted ``0.25411510467529297`` seconds. (The operating system provides this value.) They spoke the HyperText Transfer Protocol (HTTP), identified by Zeek as HTTP over TCP using TCP port 80 listening on ``31.3.245.133``. If we wanted to move beyond who talked with whom, when, for how long, and with what protocol, the second :file:`conn.log` entry offers a few more items of interest. For example, we know that ``192.168.4.76`` sent 77 bytes of data in its application layer payload, and 397 bytes in its IP layer payload. We can verify that 77 byte figure by decoding the HTTP traffic sent from ``192.168.4.76`` during this session. We use :program:`tshark`, the command line version of Wireshark, to do so. .. code-block:: console zeek@zeek:~zeek-test/json$ tshark -V -r ../../tmi1.pcap http and ip.src==192.168.4.76 .. literal-emph:: Frame 21: 143 bytes on wire (1144 bits), 143 bytes captured (1144 bits) Encapsulation type: Ethernet (1) Arrival Time: Jun 5, 2020 14:39:59.512593000 UTC [Time shift for this packet: 0.000000000 seconds] Epoch Time: 1591367999.512593000 seconds [Time delta from previous captured frame: 0.000309000 seconds] [Time delta from previous displayed frame: 0.000000000 seconds] [Time since reference or first frame: 17.461008000 seconds] Frame Number: 21 Frame Length: 143 bytes (1144 bits) Capture Length: 143 bytes (1144 bits) [Frame is marked: False] [Frame is ignored: False] [Protocols in frame: eth:ethertype:ip:tcp:http] Ethernet II, Src: 08:00:27:97:99:0d, Dst: fc:ec:da:49:e0:10 Destination: fc:ec:da:49:e0:10 Address: fc:ec:da:49:e0:10 .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Source: 08:00:27:97:99:0d Address: 08:00:27:97:99:0d .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Type: IPv4 (0x0800) Internet Protocol Version 4, **Src: 192.168.4.76, Dst: 31.3.245.133** 0100 .... = Version: 4 .... 0101 = Header Length: 20 bytes (5) Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT) 0000 00.. = Differentiated Services Codepoint: Default (0) .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0) Total Length: 129 Identification: 0xfdf1 (65009) Flags: 0x4000, Don't fragment 0... .... .... .... = Reserved bit: Not set .1.. .... .... .... = Don't fragment: Set ..0. .... .... .... = More fragments: Not set ...0 0000 0000 0000 = Fragment offset: 0 Time to live: 64 Protocol: TCP (6) Header checksum: 0x6308 [validation disabled] [Header checksum status: Unverified] **Source: 192.168.4.76** **Destination: 31.3.245.133** Transmission Control Protocol, **Src Port: 46378, Dst Port: 80**, Seq: 1, Ack: 1, **Len: 77** **Source Port: 46378** **Destination Port: 80** [Stream index: 0] **[TCP Segment Len: 77]** Sequence number: 1 (relative sequence number) [Next sequence number: 78 (relative sequence number)] Acknowledgment number: 1 (relative ack number) 1000 .... = Header Length: 32 bytes (8) Flags: 0x018 (PSH, ACK) 000. .... .... = Reserved: Not set ...0 .... .... = Nonce: Not set .... 0... .... = Congestion Window Reduced (CWR): Not set .... .0.. .... = ECN-Echo: Not set .... ..0. .... = Urgent: Not set .... ...1 .... = Acknowledgment: Set .... .... 1... = Push: Set .... .... .0.. = Reset: Not set .... .... ..0. = Syn: Not set .... .... ...0 = Fin: Not set [TCP Flags: ·······AP···] Window size value: 32 [Calculated window size: 65536] [Window size scaling factor: 2048] Checksum: 0xd9f0 [unverified] [Checksum Status: Unverified] Urgent pointer: 0 Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps TCP Option - No-Operation (NOP) Kind: No-Operation (1) TCP Option - No-Operation (NOP) Kind: No-Operation (1) TCP Option - Timestamps: TSval 3137978878, TSecr 346747623 Kind: Time Stamp Option (8) Length: 10 Timestamp value: 3137978878 Timestamp echo reply: 346747623 [SEQ/ACK analysis] [iRTT: 0.082118000 seconds] **[Bytes in flight: 77]** [Bytes sent since last PSH flag: 77] [Timestamps] [Time since first frame in this TCP stream: 0.082427000 seconds] [Time since previous frame in this TCP stream: 0.000309000 seconds] **TCP payload (77 bytes)** Hypertext Transfer Protocol **GET / HTTP/1.1\r\n** [Expert Info (Chat/Sequence): GET / HTTP/1.1\r\n] [GET / HTTP/1.1\r\n] [Severity level: Chat] [Group: Sequence] Request Method: GET Request URI: / Request Version: HTTP/1.1 **Host: testmyids.com\r\n** **User-Agent: curl/7.47.0\r\n** **Accept: */*\r\n** **\r\n** [Full request URI: http://testmyids.com/] [HTTP request 1/1] In the highlighted output, we see that :program:`tshark` notes 77 bytes of data carried by TCP from ``192.168.4.76``. I highlighted what that data was, beginning with a GET request. The ``orig_pkts`` and ``resp_pkts`` fields report the number of IP packets transferred in the respective directions. The ``orig_ip_bytes`` and ``resp_ip_bytes`` indicate the total IP packet-level byte counts, respectively. Another way to look at this TCP segment is to dump the hex contents using a different :program:`tshark` option, as shown below. .. code-block:: console zeek@zeek:~zeek-test/json$ tshark -x -r ../../tmi1.pcap http and ip.src==192.168.4.76 .. literal-emph:: 0000 fc ec da 49 e0 10 08 00 27 97 99 0d 08 00 45 00 ...I....'.....E. 0010 00 81 fd f1 40 00 40 06 63 08 c0 a8 04 4c 1f 03 ....@.@.c....L.. 0020 f5 85 b5 2a 00 50 dd e8 f3 47 b2 71 7e 69 80 18 ...*.P...G.q~i.. 0030 00 20 d9 f0 00 00 01 01 08 0a bb 09 c1 fe 14 aa . .............. 0040 f2 e7 **47 45 54 20 2f 20 48 54 54 50 2f 31 2e 31 ..GET / HTTP/1.1** 0050 **0d 0a 48 6f 73 74 3a 20 74 65 73 74 6d 79 69 64 ..Host: testmyid** 0060 **73 2e 63 6f 6d 0d 0a 55 73 65 72 2d 41 67 65 6e s.com..User-Agen** 0070 **74 3a 20 63 75 72 6c 2f 37 2e 34 37 2e 30 0d 0a t: curl/7.47.0..** 0080 **41 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 0d 0a Accept: */***.... The hexadecimal values appear on the left, and the ASCII decode appears on the right. If you count the highlighted hex values, you will find 77 of them, hence the 77 bytes of application layer data carried by TCP. The connection state field, ``conn_state``, showed that the connection terminated normally, as depicted by the ``SF`` entry. This means that, for this TCP session, both sides adopted a “graceful close” mechanism. If you remember this trace from the last chapter, you’ll remember seeing that it opened with a TCP three way handshake (SYN - SYN ACK - ACK) and terminated with a graceful close (FIN ACK - FIN ACK - ACK). Finally, the ``history`` field contains the string ``ShADadFf``. Remember that capitalized letters indicate an action by the connection originator. Lowercase letters indicate an action by the responder. This means that ``ShADadFf`` translates to the following: :: S - The originator sent a SYN segment. h - The responder sent a SYN ACK segment. A - The originator sent an ACK segment. D - The originator sent at least one segment with payload data. In this case, that was HTTP over TCP. a - The responder replied with an ACK segment. d - The responder replied with at least one segment with payload data. F - The originator sent a FIN ACK segment. f - The responder replied with a FIN ACK segment. This log entry demonstrates how Zeek is able to pack so much information into a compact representation. Understanding the First :file:`conn.log` Entry ============================================== Now let’s turn to the first :file:`conn.log` entry, reproduced below for easy reference. :: { "ts": 1591367999.305988, "uid": "CMdzit1AMNsmfAIiQc", "id.orig_h": "192.168.4.76", "id.orig_p": 36844, "id.resp_h": "192.168.4.1", "id.resp_p": 53, "proto": "udp", "service": "dns", "duration": 0.06685185432434082, "orig_bytes": 62, "resp_bytes": 141, "conn_state": "SF", "missed_bytes": 0, "history": "Dd", "orig_pkts": 2, "orig_ip_bytes": 118, "resp_pkts": 2, "resp_ip_bytes": 197, "ip_proto": 17 } For the first entry, ``192.168.4.76`` talked to ``192.168.4.1``. The log timestamp is ``1591367999.305988``, which translates as shown below, courtesy of the Unix :program:`date` command: .. code-block:: console zeek@zeek:~zeek-test/json$ date -d @"1591367999.305988" :: Fri Jun 5 14:39:59 UTC 2020 The two systems’ “conversation” only lasted ``0.06685185432434082`` seconds. (Again, such precision!) They spoke the Domain Name System (DNS) protocol, identified by Zeek as DNS over UDP using UDP port 53 listening on ``192.168.4.1``. The connection state for this conversation is listed as ``SF``, the same as the TCP version. However, UDP has no concept of state, leaving that duty to a higher level protocol. In the context of UDP, ``SF`` means that Zeek assesses the conversations as “normal establishment and termination” of the “connection.” Similarly, the ``history`` field is simply ``Dd``, indicating that each party to the conversation sent data to the other. The ``ip_proto`` Field ====================== .. versionadded:: 7.1 The numeric ``ip_proto`` field reports the `IP protocol number `_ of the connection. It relates to the ``proto`` field, but while the former represents a :zeek:type:`transport_proto` value that exclusively covers *transport* protocols Zeek knows how to parse (and ties into Zeek's :zeek:type:`port` type), the ``ip_proto`` field is always present, including for non-transport IP packet flows such as IGMP or OSPF. For example, an OSPF flow might look as follows: :: { "ts": 1098361214.420459, "uid": "C9EV8R4fN8bfSj08f", "id.orig_h": "192.168.170.2", "id.orig_p": 0, "id.resp_h": "224.0.0.6", "id.resp_p": 0, "proto": "unknown_transport", "duration": 6.437546968460083, "orig_bytes": 0, "resp_bytes": 0, "conn_state": "OTH", "local_orig": true, "local_resp": false, "missed_bytes": 0, "orig_pkts": 4, "orig_ip_bytes": 768, "resp_pkts": 0, "resp_ip_bytes": 0, "ip_proto": 89 } You can adapt this feature in several ways. Load the :doc:`/scripts/policy/protocols/conn/ip-proto-name-logging.zeek` policy script to add an ``ip_proto_name`` column with a string version of the ``ip_proto`` value. Also, you may disable the whole feature by loading the :doc:`/scripts/policy/protocols/conn/disable-unknown-ip-proto-support.zeek` script, returning :file:`conn.log` to its pre-7.1 state. Zeek's :ref:`logging framework ` supports additional customizations. The ``uid`` and Other Fields ============================ Notice that both :file:`conn.log` entries contain ``uid`` fields. These are unique identifiers assigned by Zeek that we will use to track related activity in other transaction logs. There are other fields which may appear in the :file:`conn.log`, depending on the protocol being summarized. For details on the meaning of those fields, see :zeek:see:`Conn::Info`. Conclusion ========== Zeek’s :file:`conn.log` is a foundational log that offers a great deal of information on its own. However, it becomes even more useful when it acts as the starting point for investigating related Zeek logs. We turn to that capability in the following sections.