From 3b207c35685d69c255657872683f96a77e927b14 Mon Sep 17 00:00:00 2001 From: Dominik Pataky Date: Sun, 2 May 2021 16:15:38 +0200 Subject: [PATCH 01/34] Update README --- README.md | 8 +++++--- nf-workflow.png | Bin 0 -> 19224 bytes 2 files changed, 5 insertions(+), 3 deletions(-) create mode 100644 nf-workflow.png diff --git a/README.md b/README.md index 5135098..1a35cfd 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,11 @@ # Python NetFlow/IPFIX library -This package contains libraries and tools for **NetFlow versions 1, 5 and 9, and IPFIX**. +This package contains libraries and tools for **NetFlow versions 1, 5 and 9, and IPFIX**. It is available [on PyPI as "netflow"](https://pypi.org/project/netflow/). Version 9 is the first NetFlow version using templates. Templates make dynamically sized and configured NetFlow data flowsets possible, which makes the collector's job harder. The library provides the `netflow.parse_packet()` function as the main API point (see below). By importing `netflow.v1`, `netflow.v5` or `netflow.v9` you have direct access to the respective parsing objects, but at the beginning you probably will have more success by running the reference collector (example below) and look into its code. IPFIX (IP Flow Information Export) is based on NetFlow v9 and standardized by the IETF. All related classes are contained in `netflow.ipfix`. -Copyright 2016-2020 Dominik Pataky +![Data flow diagram](nf-workflow.png) + +Copyright 2016-2021 Dominik Pataky Licensed under MIT License. See LICENSE. @@ -34,7 +36,7 @@ assert p.header.version == 5 # NetFlow v5 packet assert p.flows[0].PROTO == 1 # ICMP flow ``` -In NetFlow v9 and IPFIX, templates are used instead of a fixed set of fields (like `PROTO`). See `collector.py` on how to handle these. +In NetFlow v9 and IPFIX, templates are used instead of a fixed set of fields (like `PROTO`). See `collector.py` on how to handle these. You **must** store received templates in between exports and pass them to the parser when new packets arrive. Not storing the templates will always result in parsing failures. ## Using the collector and analyzer Since v0.9.0 the `netflow` library also includes reference implementations of a collector and an analyzer as CLI tools. diff --git a/nf-workflow.png b/nf-workflow.png new file mode 100644 index 0000000000000000000000000000000000000000..67cdd50ab79e81cfbc26ef56022d4ae21004f9dd GIT binary patch literal 19224 zcmdVCcQo8>_b)7wAflTvIzup82tf#$&E>pZod!o^#GRYrX3|f4uKnSxL-%%XRH*@6T?ZE$X3$5-Bk~F&-Wssfw}! z5)bb(0{nl8hyeUeaR)|&Wv+O9=Vy)s4J%-EWZ(xlF_^vI)(dg@S zlBZej0W8)wzozPh-G52jj9Fn=iJo)oDxUgXh7co8uq*EPvJ?tosJ0(mac{eF9&#H% z5@63+iAe7w6|!wjmC;1cSw-+vD8sP+Y;YLS^Ikj-1O&YwmK(`RB!?dd(c&JX{l9X9 z*3n|VX<$Uy2o;jozIdAcQYU>Wz4IfLYtsx@ZFIe2s&ZiA|GYHOn5J|jSH!q=(Y<)w z2U=K_$?oS2QK3Xw{j*-JQI6Ric0K)Zo-Pl^yOmeumc%_?2JAwmSf z%=zG2Sg%p2+JOn1dVIw9$&6)l zR7+SzXtVv3GY?NT)NVu ztgF4bXlb2J@pOJy^EDNIN9p-ZF*kO4cXO_r4e7gO9TW2 z;>r8azNG;+Dz!N8o76UsHk*wsgWHNURVOrF&Hbkmyr!&tmFBd1AfrKn!C>z8-DOTA zkY{H%`tE(Eh1~r;cz1bh-rlJuBLZb+*mDW&gaj(h%+B>d{%cQufzIT$X?4=pwkPhc z0>6bYF<0ZJ1G2T14yU-m8CI_xs*z{s=SM_)xkaDex*(er^#4xP<=hw~yZBX{IZY>* zPqQqXXkCiJpD-~GI?TOrd0bDF~fo)w_8@Od!dWn z`F`yXs`Yt2YVFPQ6|9>0-YWFIyh14zJEPC2|CoV<=?q`)*R9TW$(p#wpPCQ9$lz2J zu?lSJs&MvQuN#~aD>S>iyITj7$$FJeT#WYBu=9R-pr2b-C#AS;0MKY{Y6`M9oc-WX zba$pJr4i_g3+$X8s1p&h2r*MFfvawzg2gp5g_}6Dji-LwjI;~8RnK646h5i^Xd$sW zgS$QAA4qDDd6&zruC?{vy?0nX9;jTD?(G_z7%Bmo=fT03EEb409%`%f0Udt-%3yZp z%XJ*CWYq2{UYO5UAEz7HwG0%o9W;G({xejtxDqp6$;i_`RP_VWb7V!jP-SY?)fHt6 zG!%yfwJvw=HPFNEoY8Sf49L-fwGm##1{1B{!2_?POmtz&zcr!d)9VAE37oF3@}tue zzuw+*oQZr37!OuWVz{3$Vc2)V0gw;+-^v8Nwk2}m#F^}=$Md&rhj!FroT-gT z{sGQ8T@g50KKyv= z-^-Kh`}}{np;=OZndC~?bl>6DM2=GL?9zu-pQR&|GCP+$l=LimGaL zy)uXPH>#EY2}qG-o+dfp62-8LGQ4d3UFMp?20s=7lB}fw0ZXr|$W27E?C&NhZ^E?h z4<0RaY5H_b+p^3@RkY~EKD&w*9e8G67?4|NYz`dmvnNh)i#c7*P`Crg&zFW zkyczymA8y~G=$t;7`iqx)}@U83Qd5a1wsKXcrdC@OG^gYQClL*`<>Yyg=hN?i`q`L zUAMNre0b-NqO}i)*k*|!LaA6b|7nkq=iGfLy-xE=j{Wvr^ON0`S@QTU64KRYZ@j!N zEb_Q5zY3BZ*9xDeK5OmClc@D_46#8!y03AWdHP=LEkm*ABXUHAc~=86g_8_5VWOQ= z{4%l*Y;72okiIvWgt%};{O*`V2YCn^1yt7c&U3@hlgaAu?Y1owu$o}8*y}hzW>PZD z>l)M!S`bd#UUelg_*C%X8CwoI5$5jx=s|`!g??2|+yMv8^#DeEQx?s8kJ4N6+$CA5 zQVL(ez`L1^WQZ3y-WV?FycR}XiF+Azy6(L`ek^PvEfM`EU$nXDJ_~q2Q2LeSwY6-e z`wZNrimV5-bVhd@87!`|={i+?FL^(g%mOaB*BD65$=PAV#DT~I2>V})D{w!dnb=LI zV;A(eKWvfL)%_r7I7MjULy5s)XYJxZ%?k4+M{QND7eS&EyQS)X`nrdUMnqNl`B`6# z6D)At*VH6;SO~u1;$BF$0J+B{;&EU1 zaIh@qZA|0+#jga)QKKy?eGMO5dGffyUX|N^dHeklC-9%7Q=|1$Lgo_ump*=cj3$<` zHBNxkNbss@718XKyZ(IORI5q;^9RIp%lWIivw<)}Pa4DsPA^r=<#RP4O!vk0()%iH zzf_U0To+~FwW%MKRM22Y=l&9AXK1?jpW@-yASzlbY;bISOlBtY_bk<)-oVa5kUW{| zsUm#DiTER@n4MBan`y1U{`*uQl0TFuudA&Ms*>Nj{lJyDC(WD9;g_;TZb!^=816L3 zL|77e9wXrhO{a{Y;9zOyDe-XdIV;yCr2_%^5Fh5Py>0EaN*KJ~kLyk8gV&6zNNo+I z2mg_1(k-KsUBl_jGr=dP+6qc-E#l3DdO<($GXkcuLWGiGBqA`zEDcz_9t4(Q% zt|-11$$pb5rtkVbA*x)^fl(zsbtVl`9@zNpC)U$20ed?MOUm22{dlc%Vz=Bsuo($! zPt5hm{U*HSS#}9}*8||Z3Z<6pVX9D4E7Z3G>ic~vo?Trn{w>vQE9k*iW$hmcVFIMu z^@&>Vpj>73Ep>G5{(AmyU%%I)G81yJ+MF6gC(WGsX|<(!ijrDUW8qiN^&`6dr$5c@ zEQrf$ry}nLkge>HJb>P%oPLo((r8^`W@#D90QcHHdDr9OR&YsJn9Z(cSkb`}*1t%1 zc+=t9xWOGkAPU_aI3HX9SN*-=h2Q>@OsBV<^9D?LGX! z4e`q_e%QClSq3v-YH+y;XNq{r9HRc10m?wvI#ntmdQpiV@|hVe+bmywd@;d%sW6X3 z?9paCBpyO}BQ#>Yk=n)l?(KpuWNsj*bo^uMo3aQuq$S&jrqsN( zZLiZiK-M!Gzk=26E9Vry8n1(}AH1Gzb>>MVe_Nz9W5`ICf*zWGdnuOP&hgts%JmtO z3YzbX(Va(+3ezzwD=TbA-3X6O`p~J#6PU4yOLTE+hLjHfIq8!nXRH>!>4P-!m%RB~ ziLe8Z9Xwqja5}bLm3p0$Iy>^Y@h{cqSL%Q=e1ndEgGO6ptZsL}^o3ece)o5#QUaQx zh6rTt=i4j*$pH8|3H+?lJ-ow@nBH}})l)9lXU3RrG{}j_9$0c6IP)QIdp$cJ%_rt- z2OT>vlj1_~r&emGURV{C^)}d32&5mSo&1h)eMHeK6qmNq&xQxiyLhPbaL_sQ8Ji5P zz9K-QKQ^}%n8-CZXhx>^3C-;ti#+xUYn-Dk@|4Z^nxa6ml&C+RLk4yilN*^E*=zAj zxC4SIe}X9|!w5-sye@cUb>lN~_tg!ezAGJii)bE1Ej<(XK&8_iQG*<;Pn7v?+Jj7k z@TbgnIJ9c-fATOEo(d2(scb7ImX-#9fSeywrYr#jvx3J>eLEQ3jV z4*V>^F4GkhI|`hJIlMBox0|t*0Q1n^(8#XZ;tZ{h_}V`_95ILtPwFvrZ!9tSKt^y( zVC2iEK(qC3%^0M@Z>R84B->msqd#sg^|o^$I1r&pZfQb8`s!BxJ;a%C9zJ=-T0X?4 z?{eykJ}vn1!hDJ|a(6=*(PMbT+oPKdQ9pCy6q^)&Bi0chd?~80umIUk5dm@|pyhvQ zzE)Y2bbAsl6Xo@^ccLR7oVY%-Na`=Tt=g%{JG1Z|ZYRU6aQd=yU&JV5vbyT)zQZ}S z2H72SiSI^n)~iwo9Ua|`8l-k-D>ZL`z8X>isbI@iMTgoO)H9($@g+B~%R95dd6b{U zdYCrnD)V;=Pv=G!#<6*-#n+_~CTM(fT~%h}u(MJZ$<7RWrtGa5x4GC#4&(6&vQFfr^daAiN11aU4lv@b4ZOafZ8>&1NcND-GefPF| zwaYJO4a+HM+lKiAsW_FAI4XBK-Iq?y08Oqh@yU)IrA59_Z7KKXiWm6N^sc?JdH&B> zb-Wubu%YIjJkFuBW69x404D${71}nG6JQB2xo@w340?qv^E;}hsvxz`*(Y^b;mGmr zZZSB$*psJGe}3LX;Gd8w-jVdxi{yJXb+|gf4&}T|T|Ma1o=*2i^%cxC>n%lUiaacd zRlGn3KoFv~ekLjLr93I{W$^%&@(u&~F-Sd*#bnxW#)EoZ;(I zH5>a|^4jJt`ZubJJntcf7Jck}n}_xV^Rw3;Q9mj^+4I_6lcESYA)Q~9dH-9m zjmBal@GnRj3&z~I=dM+1`x25N^XXa){a|o;Ih(YXdz)}Od_gLzoqYbJxx;{1(vXhk zRQy*+?()IUikXL#1foMozRUyugOgT|+b530r#K6Kk*-M@LSo&8|9-gXqWmXGxS~GFbwo@W}gD z8l>UYY63Rzx{jaQ-PK6ZPd6>RIC7J2Z?OHId2e=bxAF6mox4Odx_cJ@9j5LwZ@K&-@l4<;eSTH zVI)D6qKyGDDF}&niBE~oPCV`p{pL1OAH&6dh>lxol8x7I7ebS%z2+8kDwZcFS*SY+ z-1lCmrCF@D1$Wglh1GMFZx#PW=lCYis_QV$X*kS$b}(+vQ!eQg27!Ts<(JdNMIOcr z`=c8UC60+y<&sEtM>yQy=L%}4YQXi__8#LpQ#^f|B2xQOURUA`lB2CVCwU|^b@Cud^BPjVLilNZXl0<4#aviv?&myIyVN1b{)_LcRc~GNukZ9ysX;X3x zmG@*QWfm5`WMb3d@<+1DIrx&lBurT9L${?e;om!ncT$cQ_?X8hkIbDE7wS-qQoud} zHJ~324GoJXCME}U9xFSn92^U?gN2W>eq>59xG#P8m+;wLRx+-8+OxI2{l&KTO>+!L zjb*tZQzD|GCf&pw@7`HosD`no-l%_TggZjVX>r=UhKcCY4gK-QPX!}xgW$Ub7;!R%d;>^co2<_jvneEk;wP< z!va^XT)8JIEPT@0*0w;!Tm4Skxb@;indOK6z=(oDu+WS{_nYfQkNJ1%0@FJ58Z8LK zqyqPCH+bTB;KXTnKa7wIe+Jq}KyX|g|3P!*z)Pt%i}p6L>4Du3^-d+ng)yCe{K?^4 z>xes&yy@UkAcA^f^LI|8T(|th&hFT@`?Zi45aUT7)h=6ti0fP%U%c{eTW{|cABykc zK;UZ|1v@O#F&;Y+Ve_!(JF$JTdONUIjtsKSwQi_`(H&~(J$rUxDXN1qreI%nuJ!54 z_X==Tpv*eAg@BGW6?F<#dpHqpm`Gm%2?fZSpwNkHqh8oJ0hu5SZ$vc4Bk;}>@?uMi zuKCK@_K<;PphWXZ`#Nw$AEo^kCodT;hRiS!h#YBnm8;wXbe$4GD~HTw(oRs#v#ABz zu=}eg%`Q26PW;&phtI)=|0r1eC6vSKw_JEnWwm$AZS}5h+{uVL z*&^Agm=`)4KL@d;If}pi#Xz|Gx8&XZ(aav1=AWa?$HzV{o|Q^=>pCDHMi)^ok5tk< zcz`iC{z-NKlnTQe8M~q)N=N4dwvlnzv*_e?|6lfP{!5GP1OqHIlSA^>&jCP(Nach%l_+n z3Sr@j@V(WazQvRbJ`u&md>-o~+brdJ>0MCw^R7nh4`_wiRdTB_p^i3;5c?fx0b@`p z0kXOLuOoIBDKxID)FIL`*GLx+=2TCefJ7i;7YyjgJ22C@q@Ya-+-Uj@|( zV}VclQGhPEL!9^RvT^y3O7XR^O6AHu5|tXZ~*ReTw2fsfVJVo&jX>kTNf_tZ!Jj z*PG$%?NsTfJn4{>loYGnyu8GmxmU5R5#+6{t?W6#`0wIg5D;+Am7J25^}_#TfA(av zF>r6xCTnqgeEb#)qyNMwtKeD!j94X}m?ssoeHSAXQx%NOv^7GnudfFf)VM9G-4T!e z{xUf^!gFn?d1B(UCNc4+LCgkp542F&;K)lLaBfjXdsBlfqPMFnGPD4yQ)ZH(TZp_l zGBUFJbAjww0c3$32p(4WoF-I-5`?kv+js7q-4ziL(HyvpEm>Gv$_czU|5030BKE|A z%by;|FadTW4zkUK$&D0)+r!;Zn7vY?njOEC2bBm8sxwC`tD9nig5IV1`3GYI1CRC| zeYw(kdi2NLez>$G+5$v6|7~F}Kg3}*zJLE7_&POJdT)O}fC>sdyvN5!5_o>ZWr0Gu zZca&A+Sss*iB%c3&y7~t*p7XKhsv@2>P~rNSqq?aq|(u_rg8;EL{1(Er;17#Y^cCh zL2x2dMjuiuBHPjm)X;q0Jw3E_H8puv_QM;C-!mk$A94@E@YjY)@Oj1Q;8smRmslQu zuXG%rSzC)fJ@s4o^@}DY7LDe4oW{h&6rY}c+P9V%9zsYiT~S^>`Yx79N8|6)zK?9n z5=?+iExgQAW~De?SYP+!6%pB8TpKR?DRBAOvuEDhb5~93y)R`yj9gf-toPo&%wjl< zMMP6DG8Vhlilj^V(l9eKXK8nI_LrG9I=H+4DPO(IpM3W}#e4*T+abYm=bM|GYkpHB z0unesKXG=dwfdGcQsd5LH&h%Y>Ntjn_xb1-gPy*=EC&@EH#ZS@|Hx;5$@SmA&F$=n zxwyC-CaSfbNN{lx@*7sR0T{Z3N6sRxU}nY&9tcjDlA8LFgRde68hqFN*Q+y@zzCkl z@9HfuDFvImM%Mqi<6Lc|c8S5KnV_JcI-uZ-JoxP@S{V|aaR~{}X>Xa};lW_AxWvRT zfCA#;;%2~b@Pfjw)AAV8`FPd7hX9nges*%8q@zPqYSMtV`H^|!iJFNCv*m3%r>S~< zf{TxqLnVfU<$Cpc8C=V^pZ|js=b`bCZB7<`1n{jn08JAfZEyz7<=IZvKjTRzCn0GA zmWk(A4QcS%BTnSi?f7Irtm^PBTRsAutYNLpPE@_$5tWaR&$HEgvdNVHla#sP3CbKO zoA2oOGa@OaIf;p>VtYa#Uv>eFYxm7Q)&2*`=n9$-3FqRdiL=-sQ`e zr47&~pZ%`kiGO{NyVVY!wmELrw2QcTntcaG$}iLPF-37 zy(%s#d65GyKA#`^WG^TvsI0qmt(N3<2Xr8DR+(r{9ta-7@Yk9nA^MT+bF`^6mpr%E zM}>q%moL}yHM0UXcRF9NbOIw$dp!If(3JEJLw-|A zis^RBdJbV3e-xKyCMYl7Qp(_3y8T8N zb0j7Epn-K~EK}L-j@oyB7K*J`g*u#Y^41~%-awoKaT;ro7b(;cDrf#wpA7G$6q1sf z3KJ25c&-ixhlJn{m6?*`L8bk~L2Jy+#wJ1|L*f!16*cvCJFV$`SiIRxQ?U2Z&P_ZJ zEMEaIXFFcWkBNyX_gvE%86R&Q9E50SXmow~(hLq}c46U)u&}U*-5^nw^E3fo{j)y= zKp~$eCf)!*mLy&bCtp3H>VoBk3Y_gzuXM?C?p_2He6=fnI`5;K|v9;J>Q9s2fS`f zt>+pHoEkvfwhT!x)#5c{0i$X>B_*X#j#=N5Z#4>uihj=f{&I&4W^uJ_(hEdNk9nb! zx7iO#|7)>ORCjZ8c41lBCiSPi-7szqw17!{mpTJIJy1Tphe_Cx5j`L-_is21Fbz1m z8FU%Gb8w*hZpj$V(_Pkhj&_=?y@Z!4<{S%$$T&DRvXc!y=k-+q4ORs3V6v%c zW)A??R6$9n(l*iqxNRo`RtlG@N@BG>@VS|p8D7c*0`MnF;W-G}E^)_Un?neBfBq?j z{GP0%4G0LZov0QvGBWD@WUq@CR97cS#(b~fqmzo~4Ne&umY?q)KT44B1YYxWu)lvt z0@W=Gv>hp@i;hGS{7XEqQXps100^Z_8UxNSlXYH2d9AlJ(8T0z26|`*N5{KxbW~Ju zQIUPQRd{HqSyvpJYU7P0EFn2_Q*STDmoH!Ng7Vao-H?OD`ohJART@tw6IOY;fiY`U zIZYl2=+aYBQ9Z1%?s<^%KvX${%l3}!4+y0nAcp)c3Ip_1zcM$3nUd(rEwIDH8s7t_ zdT*^-O?rO}y#Ap zcJz&#LwiZOWhSz)_!$sQvDDE?r#>#yk*cVyoWjO36<9AzBCfCi4RHlQSjJ8gRw5}W zX~}1Q{pj$^%iGV$(jGQr=!D(uz_E@IzW+vbjJ?w=2Ed&$0Tz%yk=xbX?cjZOyf-k= z-!Jl;@J12{GCDz0oiphEPKQBcpYE@V)B9Dm=6-USVP+jxr4nHL8a2vVV=KwO_9fIIAsw z$J5dlMO|+BCBdzzjL*f>v!NMURQ&zr0oR)$jZjt*jE1V#j{qsQcvXr({@4@8rE;c% zS%dk8HRf2UQ2q!}_kruW7e}jb1d>&GIF+vo{ zTWdL^+Jko!d66$pybRlPyuE8%+XkrHhsYJr-8d@p-< z9H=}l90H%?;XVC5c2!JF3>U7kv9Uo#Y$;ApPiKDp3I{RMJdg}tP<{Qq)2&wW_vPim z85tSn=m{%n|1F1lBE~{4` zPK6GI-u~6-CjD`MQP3hN!Gq7}Ot_xjhi&`UT>&@5k*SJatt)(e-BGJCzDhBVdwDGz z@O&T`A-|t&cmht$^zpFIzfJ8W?Q!=dIAtkywdmUpcIQv*J3q-SSQDG}du08C`UV#Q z&t-i?2!tf_D+Bp1PoFAUT5<$ku8CL8^WK`dg!jg<%IUSR4O!6hONI;}KwV#V0Q!dq zPF>t%nW8FF0oFX#;Jdy1r1`oB2#1li$iaL~77WO2L0<6swiU6&({-XZ_t79X67pQt zwzRThyLF2^Gc!}%XO|nq(;?N>;voG?{>jM`4<0maj`1(VX$nX!x2|O*VrQwtTOZhl zqpm4M{}%CLoQ(ZpA4nFj7aC+@9PhDqPmVT1&I}f$%=g^bQD!Y~16-=C2*Z(;8WLHLx4IEWrxc`a@aTR>|wM!g7X zQV6>4^|I#nQrYVlKGJ*DE1_%Kg12wWY42Qfxcl*{(i}TQZ&6W4uI*rzaIerC+J4A| ze|}fTcV_9&0gD0%zp+MNjrH)T!YJg~PsVp_u#NXas_|JbUi^@;`dJ7pP@aRzHKdg0 z+`{8vV{-QQ@0d4lSU0NT;wW~P`$E#DKH1YAh~Yu*Ia`9B8b~1(l$2TlZm$fP04o-@@IOl)g6!EYyTVSx9tOMXDIb zNPyKj9mR!J%&aXHeHa)7U8I_@yWRZVv&exw<^0iHE9y`Ivc;hkLC1R%2g{i1qH>&; z^_TpCN}G$n{gFn%YtL&`oramoZie15QSw<|?5llB-=*lGM>ZGFS7W~mQlnM*UJ8Y;etOq`?r(EZI{BHv ziP`D>D}9)^t)RoY!~1|?iIPOvul04t8z3*|%&2`gZgX^WWII?u3ChR8z*)OBMudjG zk&hso*abiJ)<1K{3uXd3qB$p7&?4_d(T5cgd9c$apfgR zNd{L}*L1&nUPNDCpZA|%q(Eb1B|KMIVK6M8sAkme+VCa~s<=o(b0UifFqXJHOwa-o z>fse>!ipMjx`l@tOD;>}*v}avCX*f-Kx=94^P4+BA&~->b z`xAe)#BJ+-TTGS&vs^sIk`o66eqgBz;cKHAarKZ)H%do$bHV2h&J%f&<)<%9fj5qG z+kCF0Mp(t9u1FEGvP~eHy9&8 z=603_d72}?h+F60J2G}(ZnP5Arw z?~T~mbCN47DoM zp(O6x;z(nl42^(M^V*_2w ztE#3Q9UU_&oF*kE8hj6O7FdOasUPKQ_|MMG?YysczOqe8PX2-fA3r1+?0J-vlN0^! zanZSzot;m*P-UEu=yRbVqe;oa^ndDM%5`!;HEfaSsX17pD(&mnuPr}0O&(7HUttm! z5y8&GL!5leRME#r0#vM@x3wuajD5UzorW(cI$A9$BV+Pz$H;?@*H7|P6L-Fy{M_H^ z03~%gRI#1@?<=H$lJ{5PI{qmbK$z|$fb%>s?ie9rU`UI(ox<1N(lS^2klPOiA>K69 z*Uz{S-F^Z^vD2YUHj9^>Y-|n`2Ii1Dg9Tdiu0j_nn*Y>NbB%Fn^hb5H^wl)_J9d^& z49o`AfhJ_)%0^SwR)7-p-;Ndu31|8A%J z2Gq;ORx!8*M##QU``t)6D&lR!+riNAaA{iNuwgn>qDX_#w}15(l+hnS=6FiRf;(iM zsP;K_kdAA9_~5Y=I~LEbG^J-?@C(%E&vREu^}he-$`5%E&MOZ=WqL{2=KJZ=(VwN% zr_71zcijmeo>0gQ3k6VgzS!T{`D#$>0SA7; zbOQz&R6@I((-(!HdBl|c&^u1(SI$Zd_G%pD1{IEZ=m2Frsg($`kr ztatmuFy=aiN?%|ca ztRP9Cf-@$dX=DO=&!1tS#2g_mBe0TpXsXOdUuEcpKA9N)~fF~5<#IkISM6@YtS?3d=D)w!~=JM9v>jk zy=#*&6fJv^5!fxSu1omw$!J{evdljzHda;3oL;!&t%06iI<9x~2NgkjMrHi?MCHt)?%XlPWxI7`45O>Bj`V3&Nzw6gi&!W zJ6?LK3V?@T8-?6`ngK`yuXiT#JRHVJx>TB`uSv32Qp|OiPj88^(*v%-<-6CGZyYA} zCt?#;9e@(84!Ysa25@3i!S+Nr5wCWl-U6(YI!XS4+!eS(hq{C*caDjP8BR@0%lh!) zgM__>1y)d0^t-5_V3((hi}LXBaG;f~?ct02tlqX3770$~=B}!&6b|N?9y83j_*dCNqhVDyXSzI z*nMa&c)YuH3)xwR)JA6Trww=3$7xv!$a5fAI1!NFXD~i=AwOMISCo-nRhSJ7) zkVTu$)~*8L1K-&P7v_S`L&k0#HF$Xi=J6tK{?wl;JAVn$@{q!H`T$4wD^&tEkQ#6i zi;E2AelSJHfkk==FtveG!m&g)2QKA=?>Bu@vY}-8L0fo-FW0Q9H}%MF1x!$jSr04W9V>a4+MIT3q+*Cq>?#h}@(R{Szo8IZ;QF8fm_ zfIuv2P$6%O8{zz6PSCxdLc5~kIgl5hf*haeAwJ-q0#*a-n@4KEyjnmw3-G z#iIXoA9448G3n5Y*eL~kfGR9IaxnM?;}5%#e>HCa|6MByV}()Y-BeWp40;I|V&H#y zKomykE%&CX=VLSwR`!$mTda^&v+o#6dS`N#cq(EYmwQE|39!F=V;l$tec>!Xb?dRJ z6nOHsRJE^?)EsbQ6#|~te`S+BgfhipzZcJl>)A!umz%-a)Vs1I% z?!!K#MxSlLrgIjN+AnS}HrfF`^j{4p+?q`$tXJzScJHA0B82uaLi_0r>`G%C2!KEP zPDj_k>5E|$Ep}NBTLwtMX9NUZEkH5K{pXEFbc=`|J-P%?D&GBogL(2^_`d0#urTFF z8uhi=?|C8jHJbJ8GR8o~J_;X4f5?APikNz-hXGW2z&E;M%;d7u%i}a^{n=!Byn_Fp zwAb^_B@JcqBgx^L=>hjoq*E^R1Sj~7^lQ1$lyqEO5)9|qB3CRM4L}qT;EF>|Sb2eoTg0!f_ zyL*l2+guGK3Mj(}(3mV@9fGQEV$ihstKp2ZZ)eNm8UDH3GTJZ`XU zMa9RjhWyi(Bkv%-9)9r5vFPOW^%`wKdd&SuRMjn+(xp*}_W>c=U)%^V`LPVs$CMlG z$C5N4((;x8aZFtDacjq8;dnsM*~nuGzvpB~376Bj41m-e5Uebxqi|TgA!_7`{OEM# zfn(~FRRC;@l@(j$(}vYm|-vA-}=y_@%u2@X6~ z|88KZG7u~e2M40|1{m4$N(DJx9wtWx_f;7Y?km2(zJjfad}70el>jWv+5VpQ$uA_u z#k)A(eY8X_H#o(6_I2+*{kgSq>4~u_u<{W-wVGMbzm*1X_E!uGqHb-yK{mu)FU>hX zQmlW41;QEm?7Bh7@9THrHqrPrG;M18#OPCFrY2G$6TEY7QdGyo`D8MASwWAk?XnHJwp{rC99wX1?-K0%_F{J)(|eR$^o;K&KDzF}f^ zU7HkeAyFH}@qqtG9I*=c!2+)gMTL&Z{bpAPfVSg@NX`P%B=B?Kjjb~Q#xuj+41n=H z3;s!2iN~pc-BTS3BFxN!ZFCntF9kylvx%p@9Jh!psgfk)bR#2(jkaVb-{j{L0xOmH zWKXuh9q0vL@err?8Gat7Ow?#SRUKen*1u1jaZU)dAIDHn<7^Z&Z_dyHQ6P@$~^9%Oz1! zmfqZK&QK#{dx?&Ro)K~(O3vtK0M8T09QF5?B4HUM1&jP5Uxd@Y1;_+}yT|+HjjgL) z{@w%uJ@vTx2rdM06I_DE5{Kp{UEIL`7bQQ?szrJ%^~P>xdLi_&PgOrOK||*3LXjz`a6(yD!8t`N?can?F2t<;1w-7= z&jydv0Z$o-J(@7T8QLmyaNdHW2R7Z$;+2PqD&-J(io@wA%C75noQ~!Qy>adYvt2i> zuh)8Pg=n6Q6>Kh&m4|ILAV1ptc^Wt;fQLuiiTk|(n&6Alk3Ed<#%Ff>J|QY;-Kbb8 zptJkafeDZs&vTXQcHl@$J5COG{|*RtZX|t3z*bdZr??Qme!cDb5laYy|AQU2jd#e| z+pX5IZE@$uXh+R&zxEOp3w3WI03bh>u4cMz^%{8;mS@}3UkK)Q@{_J7orzEaUjqn@ zH{l;X#oypXfbl7A<2)bVyk*6&pJC(Cxdq-hUCv9yg4yaOsC@sP1Ly+&1vh;kAwnHn z<7ER|V3h1L%fTHlwY5nbG+=E4US*K(3zW;s1ldG`-*Qgw{}#l@ee-O+4le_94F}^q zU=tA;B_zOMg@T`;V%X#a(9m6cE|nHI$5FN=j1ai9Uha9uh@*t>gvoGAMJph81OBzP z0oskGt`R7hTn2r{zu?B!01XKD_op)*%`Gf2$0i%qZUJ89RvV+I)J7SAqj(=I3SP+^ z$qDbD!<`+CdO$voi#}oWuBfCQ4GKzyeNsPHc5JMk2 zh=Jrib4KY-lDQ{`$?{Rkb5mJ-O^55%JiEEh^gG_;1Xn`Lwa$IBPy4BSxyn>zUq;6I z0;FX1@`e#dZ$KH~?LE*Mrn2aIeew0^A)H>&0*^C7W9}_&={cPVX+xa%A4b|5>}eUq zlXUet-2JVKqw?pM+D;kPvse73Ai^z4xZfo3#GNKbw{ceDCIa}%AqT?}55C>?Vd_>c z5+%^+TEF+1`mtk?#l{Cgr>Quhkec6_9!2;4!ZCE1%rhT1D-q{O5^5XuP@{PC`#ksF zrmA4Yi}}uVQ=FIndph;22|8R8N+z657kiU2%t%Wc{q+AyJ5H@HtnLd`#E!!9vDNzz z$sMEo13UsKr7kjpH4c8+qu6Y`pGd{a8r|rP- za+5=L`aVlvVPK+KzJ6ccvb<9kzuw&}YV%FSsn)R$ z>BDrgYlan5nS?D%FEdaJQKmzD=1k^^3c22)CcVp?X% zMmsI(=lh_c2ik|IsWNvqUC6ySN2`A~SmGdg*12X~ubY8u0LEgzOP*7`sKJ52BJ)&V zpQ#Zopf9H6)@16lzV2%RQG-_Gh>v$P@+9VJ2N138)Y~JjX^El&?Te=y6k-LeLD*#)dU{dq$M|GA4@WuQs7Rp~%|@?|A1zBLD^K_ArZ3dG3eOeR z_y_t<*4Nc!Hp-ZqHu~2{2OMwBbx)p6?rrUwL7H1Ov$AewOdiLak#n8d`pY@HLK^Zg zupq&lf^9Rkl>QHwWcdHw(8 Date: Sun, 14 Nov 2021 17:53:27 +0100 Subject: [PATCH 02/34] Release v0.11.0; adds Netflow v9 option headers --- .github/workflows/run_tests.yml | 2 +- setup.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/run_tests.yml b/.github/workflows/run_tests.yml index 1e63bf2..4687276 100644 --- a/.github/workflows/run_tests.yml +++ b/.github/workflows/run_tests.yml @@ -18,7 +18,7 @@ jobs: - name: Set up Python 3.5.3 uses: gabrielfalcao/pyenv-action@v7 with: - default: '3.5.3' # Debian Buster (stable) + default: '3.5.3' # Debian Stretch (oldoldstable) - name: Run Python unittests run: python3 -m unittest diff --git a/setup.py b/setup.py index fb2438f..b76f436 100644 --- a/setup.py +++ b/setup.py @@ -7,7 +7,7 @@ setup( name='netflow', - version='0.10.5', + version='0.11.0', description='NetFlow v1, v5, v9 and IPFIX tool suite implemented in Python 3', long_description=long_description, long_description_content_type='text/markdown', From b8e911a40af83a93b02d8dbdd87854510d00a623 Mon Sep 17 00:00:00 2001 From: Vitali Sepetnitsky Date: Wed, 16 Feb 2022 18:39:15 +0200 Subject: [PATCH 03/34] avoid infinite loop in V9ExportPacket's constructor --- netflow/v9.py | 12 ++++++++++++ tests/lib.py | 30 +++++++++++++++++++++++++++++- tests/test_netflow.py | 6 +++++- 3 files changed, 46 insertions(+), 2 deletions(-) diff --git a/netflow/v9.py b/netflow/v9.py index 348deb8..6be32a1 100644 --- a/netflow/v9.py +++ b/netflow/v9.py @@ -499,6 +499,8 @@ def __init__(self, data: bytes, templates: dict): if id_ not in self._templates: self._new_templates = True self._templates[id_] = template + if tfs.length == 0: + break offset += tfs.length continue @@ -510,6 +512,8 @@ def __init__(self, data: bytes, templates: dict): self._new_templates = True self._templates[id_] = template offset += otfs.flowset_length + if otfs.flowset_length == 0: + break continue # Data / option flowsets @@ -518,6 +522,8 @@ def __init__(self, data: bytes, templates: dict): # Could not be parsed, continue to check for templates skipped_flowsets_offsets.append(offset) offset += flowset_length + if flowset_length == 0: + break continue matched_template = self._templates[flowset_id] @@ -525,11 +531,17 @@ def __init__(self, data: bytes, templates: dict): if isinstance(matched_template, V9TemplateRecord): dfs = V9DataFlowSet(data[offset:], matched_template) self._flows += dfs.flows + if dfs.length == 0: + break offset += dfs.length elif isinstance(matched_template, V9OptionsTemplateRecord): odfs = V9OptionsDataFlowset(data[offset:], matched_template) self._options += odfs.option_data_records + if odfs.length == 0: + break + if odfs.length == 0: + break offset += odfs.length else: diff --git a/tests/lib.py b/tests/lib.py index 5a06bda..0fdbb61 100644 --- a/tests/lib.py +++ b/tests/lib.py @@ -185,9 +185,37 @@ def single_packet(pkts): "0000bfee0000002f00000000000000000050ae56061b04007f0000017f000001fb3ddbb3fb3c1a18" "00000982000000270000000000000000ae560050061b04007f0000017f000001fb3ddbb3fb3c1a18" "0000130e0000001200000000000000000050e820061b04007f0000017f000001fb3ddbb3fb3c1a18" - "0000059c000000140000000000000000e8200050061b0400" + "0000059c000000140000000000000000e8200050061b0400", ] +PACKET_V9_WITH_PL = ( + "000900057b72e830620b717d78cf34e30102000001040048000000000000006e0000000101000000" + "000a20076a06065c0800000d6b15c80000000b7b72e4487b72e448080000000000000438bf6401c7" + "65ad1e0d6b15c8000000000001040048000000000000006700000001110000c951ac180b0306065c" + "080035010000010000000b7b72e4487b72e448000000000000000443177b01c765ada501000001c3" + "9c00350001040048000000000000004a000000010100000000ac19bc3206065c080000287048cd00" + "00000b7b72e8307b72e83008000000000000048f071a01c765ae42287048cd000000000001040048" + "000000000000004600000001060002cbef0a30681f06065c0801bb142a49180000000b7b72e8307b" + "72e8300000000000000004801c7801c765ae1d142a49185a2b01bb00010400480000000000000046" + "00000001060002fe800a2f601206065c0801bb142a49180000000b7b72e8307b72e8300000000000" + "0000040806b001c765ae28142a4918d4b501bb000000000000000000000000000000000000000000" + "00000000000000000000000000000000000000000000000000000000000000000000000000000000" + "00000000000000000000000000000000000000000000000000000000000000000000000000000000" + "00000000000000000000000000000000000000000000000000000000000000000000000000000000" + "00000000000000000000000000000000000000000000000000000000000000000000000000000000" + "00000000000000000000000000000000000000000000000000000000000000000000000000000000" + "00000000000000000000000000000000000000000000000000000000000000000000000000000000" + "00000000000000000000000000000000000000000000000000000000000000000000000000000000" + "00000000000000000000000000000000000000000000000000000000000000000000000000000000" + "00000000000000000000000000000000000000000000000000000000000000000000000000000000" + "00000000000000000000000000000000000000000000000000000000000000000000000000000000" + "00000000000000000000000000000000000000000000000000000000000000000000000000000000" + "00000000000000000000000000000000000000000000000000000000000000000000000000000000" + "00000000000000000000000000000000000000000000000000000000000000000000000000000000" + "00000000000000000000000000000000000000000000000000000000000000000000000000000000" + "00000000000000000000000000000000000000000000000000000000000000000000000000000000" +) + # Example export for IPFIX (v10) with 4 templates, 1 option template and 8 data flow sets PACKET_IPFIX_TEMPLATE = "000a05202d45a4700000001300000000000200400400000e00080004000c00040016000400150004" \ "0001000400020004000a0004000e000400070002000b00020004000100060001003c000100050001" \ diff --git a/tests/test_netflow.py b/tests/test_netflow.py index c145bef..a89bef2 100755 --- a/tests/test_netflow.py +++ b/tests/test_netflow.py @@ -13,7 +13,7 @@ import unittest from tests.lib import send_recv_packets, NUM_PACKETS, \ - PACKET_INVALID, PACKET_V1, PACKET_V5, \ + PACKET_INVALID, PACKET_V1, PACKET_V5, PACKET_V9_WITH_PL, \ PACKET_V9_TEMPLATE, PACKET_V9_TEMPLATE_MIXED, PACKETS_V9 @@ -119,6 +119,10 @@ def test_recv_v9_packet(self): pkts, _, _ = send_recv_packets([PACKETS_V9[0]]) self.assertEqual(len(pkts), 0) # no export is parsed due to missing template + # send an invalid packet with zero bytes, must fail to parse + pkts, _, _ = send_recv_packets([PACKET_V9_WITH_PL]) + self.assertEqual(len(pkts), 0) # no export is parsed due to missing template + # send packet with two templates and eight flows, should parse correctly since the templates are known pkts, _, _ = send_recv_packets([PACKET_V9_TEMPLATE]) self.assertEqual(len(pkts), 1) From 8b5675913decf25ef6278a9111a1b1a7311d8863 Mon Sep 17 00:00:00 2001 From: Dominik Pataky Date: Mon, 25 Apr 2022 20:26:04 +0200 Subject: [PATCH 04/34] Small changes to PR #37 preventing infinite loops; bump version Closes #37 --- netflow/v9.py | 2 -- setup.py | 2 +- tests/lib.py | 2 +- tests/test_netflow.py | 4 ++-- 4 files changed, 4 insertions(+), 6 deletions(-) diff --git a/netflow/v9.py b/netflow/v9.py index 6be32a1..d197dd0 100644 --- a/netflow/v9.py +++ b/netflow/v9.py @@ -538,8 +538,6 @@ def __init__(self, data: bytes, templates: dict): elif isinstance(matched_template, V9OptionsTemplateRecord): odfs = V9OptionsDataFlowset(data[offset:], matched_template) self._options += odfs.option_data_records - if odfs.length == 0: - break if odfs.length == 0: break offset += odfs.length diff --git a/setup.py b/setup.py index b76f436..814a137 100644 --- a/setup.py +++ b/setup.py @@ -7,7 +7,7 @@ setup( name='netflow', - version='0.11.0', + version='0.11.1', description='NetFlow v1, v5, v9 and IPFIX tool suite implemented in Python 3', long_description=long_description, long_description_content_type='text/markdown', diff --git a/tests/lib.py b/tests/lib.py index 0fdbb61..7db164a 100644 --- a/tests/lib.py +++ b/tests/lib.py @@ -188,7 +188,7 @@ def single_packet(pkts): "0000059c000000140000000000000000e8200050061b0400", ] -PACKET_V9_WITH_PL = ( +PACKET_V9_WITH_ZEROS = ( "000900057b72e830620b717d78cf34e30102000001040048000000000000006e0000000101000000" "000a20076a06065c0800000d6b15c80000000b7b72e4487b72e448080000000000000438bf6401c7" "65ad1e0d6b15c8000000000001040048000000000000006700000001110000c951ac180b0306065c" diff --git a/tests/test_netflow.py b/tests/test_netflow.py index a89bef2..3949687 100755 --- a/tests/test_netflow.py +++ b/tests/test_netflow.py @@ -13,7 +13,7 @@ import unittest from tests.lib import send_recv_packets, NUM_PACKETS, \ - PACKET_INVALID, PACKET_V1, PACKET_V5, PACKET_V9_WITH_PL, \ + PACKET_INVALID, PACKET_V1, PACKET_V5, PACKET_V9_WITH_ZEROS, \ PACKET_V9_TEMPLATE, PACKET_V9_TEMPLATE_MIXED, PACKETS_V9 @@ -120,7 +120,7 @@ def test_recv_v9_packet(self): self.assertEqual(len(pkts), 0) # no export is parsed due to missing template # send an invalid packet with zero bytes, must fail to parse - pkts, _, _ = send_recv_packets([PACKET_V9_WITH_PL]) + pkts, _, _ = send_recv_packets([PACKET_V9_WITH_ZEROS]) self.assertEqual(len(pkts), 0) # no export is parsed due to missing template # send packet with two templates and eight flows, should parse correctly since the templates are known From ef99464fc5b3f516dc87e196859ba7aaa1a6329c Mon Sep 17 00:00:00 2001 From: Anze Date: Sun, 23 Feb 2020 12:27:10 +0100 Subject: [PATCH 05/34] Performance improvement: when checking if a field contains an IP address, compare the keys (which are integers) instead of values (strings) --- netflow/v9.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/netflow/v9.py b/netflow/v9.py index d197dd0..22e04d2 100644 --- a/netflow/v9.py +++ b/netflow/v9.py @@ -21,6 +21,10 @@ "V9TemplateFlowSet", "V9TemplateNotRecognized", "V9TemplateRecord", "V9OptionsTemplateFlowSet", "V9OptionsTemplateRecord", "V9OptionsDataRecord"] + +FIELD_TYPES_CONTAINING_IP = [8, 12, 27, 28] + + V9_FIELD_TYPES = { 0: 'UNKNOWN_FIELD_TYPE', # fallback for unknown field types @@ -221,7 +225,7 @@ def __init__(self, data, template): # Special handling of IP addresses to convert integers to strings to not lose precision in dump # TODO: might only be needed for IPv6 - if fkey in ["IPV4_SRC_ADDR", "IPV4_DST_ADDR", "IPV6_SRC_ADDR", "IPV6_DST_ADDR"]: + if field.field_type in FIELD_TYPES_CONTAINING_IP: try: ip = ipaddress.ip_address(fdata) except ValueError: From b10dc5faeffc2efaa38aea6c0cf1783321c4fb33 Mon Sep 17 00:00:00 2001 From: Anze Date: Sun, 8 May 2022 17:52:51 +0200 Subject: [PATCH 06/34] Performance improvement: rearrange code so that instead of converting IP addresses to integers first, we construct them from bytes directly --- netflow/v9.py | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/netflow/v9.py b/netflow/v9.py index 22e04d2..1ad90fe 100644 --- a/netflow/v9.py +++ b/netflow/v9.py @@ -218,21 +218,20 @@ def __init__(self, data, template): # The length of the value byte slice is defined in the template dataslice = data[offset:offset + flen] - # Better solution than struct.unpack with variable field length - fdata = 0 - for idx, byte in enumerate(reversed(bytearray(dataslice))): - fdata += byte << (idx * 8) - # Special handling of IP addresses to convert integers to strings to not lose precision in dump # TODO: might only be needed for IPv6 if field.field_type in FIELD_TYPES_CONTAINING_IP: try: - ip = ipaddress.ip_address(fdata) + ip = ipaddress.ip_address(dataslice) except ValueError: print("IP address could not be parsed: {}".format(fdata)) continue new_record.data[fkey] = ip.compressed else: + # Better solution than struct.unpack with variable field length + fdata = 0 + for idx, byte in enumerate(reversed(bytearray(dataslice))): + fdata += byte << (idx * 8) new_record.data[fkey] = fdata offset += flen From 77da7b16b616c82a98ec12558fa2f315ed25c785 Mon Sep 17 00:00:00 2001 From: Anze Date: Sun, 8 May 2022 17:54:05 +0200 Subject: [PATCH 07/34] Performance improvement: use struct.unpack instead of manually constructing bytes when possible --- netflow/v9.py | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/netflow/v9.py b/netflow/v9.py index 1ad90fe..8e2f3bc 100644 --- a/netflow/v9.py +++ b/netflow/v9.py @@ -228,11 +228,19 @@ def __init__(self, data, template): continue new_record.data[fkey] = ip.compressed else: - # Better solution than struct.unpack with variable field length - fdata = 0 - for idx, byte in enumerate(reversed(bytearray(dataslice))): - fdata += byte << (idx * 8) - new_record.data[fkey] = fdata + # For performance reasons, we use struct.unpack for known lengths: + if flen == 4: + new_record.data[fkey], = struct.unpack('!L', dataslice) + elif flen == 2: + new_record.data[fkey], = struct.unpack('!H', dataslice) + elif flen == 1: + new_record.data[fkey], = struct.unpack('!B', dataslice) + else: + # Caveat: this code assumes little-endian system (like x86) + fdata = 0 + for idx, byte in enumerate(reversed(bytearray(dataslice))): + fdata += byte << (idx * 8) + new_record.data[fkey] = fdata offset += flen From c12507343b477debe6b741dd5a7b5815ee95ae72 Mon Sep 17 00:00:00 2001 From: Anze Date: Sun, 23 Feb 2020 14:38:45 +0100 Subject: [PATCH 08/34] Performance improvement: no need to copy a part of the buffer when using struct.unpack_from() --- netflow/v9.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/netflow/v9.py b/netflow/v9.py index 8e2f3bc..912f0a1 100644 --- a/netflow/v9.py +++ b/netflow/v9.py @@ -212,15 +212,14 @@ def __init__(self, data, template): new_record = V9DataRecord() for field in template.fields: + # The length of the value byte slice is defined in the template flen = field.field_length fkey = V9_FIELD_TYPES[field.field_type] - # The length of the value byte slice is defined in the template - dataslice = data[offset:offset + flen] - # Special handling of IP addresses to convert integers to strings to not lose precision in dump # TODO: might only be needed for IPv6 if field.field_type in FIELD_TYPES_CONTAINING_IP: + dataslice = data[offset:offset+flen] try: ip = ipaddress.ip_address(dataslice) except ValueError: @@ -230,12 +229,13 @@ def __init__(self, data, template): else: # For performance reasons, we use struct.unpack for known lengths: if flen == 4: - new_record.data[fkey], = struct.unpack('!L', dataslice) + new_record.data[fkey], = struct.unpack_from('!L', data, offset) elif flen == 2: - new_record.data[fkey], = struct.unpack('!H', dataslice) + new_record.data[fkey], = struct.unpack_from('!H', data, offset) elif flen == 1: - new_record.data[fkey], = struct.unpack('!B', dataslice) + new_record.data[fkey], = struct.unpack_from('!B', data, offset) else: + dataslice = data[offset:offset+flen] # Caveat: this code assumes little-endian system (like x86) fdata = 0 for idx, byte in enumerate(reversed(bytearray(dataslice))): From 1bffe3a2a342b24714ca08142871c0743617474f Mon Sep 17 00:00:00 2001 From: Anze Date: Sun, 8 May 2022 18:31:06 +0200 Subject: [PATCH 09/34] Performance improvement: rearrange netflow v9 packet parsing (use struct.unpack to extract all of the values at once) --- netflow/v9.py | 50 ++++++++++++++++++++++++++++++-------------------- 1 file changed, 30 insertions(+), 20 deletions(-) diff --git a/netflow/v9.py b/netflow/v9.py index 912f0a1..4723313 100644 --- a/netflow/v9.py +++ b/netflow/v9.py @@ -208,39 +208,49 @@ def __init__(self, data, template): # As the field lengths are variable V9 has padding to next 32 Bit padding_size = 4 - (self.length % 4) # 4 Byte + # For performance reasons, we use struct.unpack to get individual values. Here + # we prepare the format string for parsing it: + struct_format = '!' + struct_len = 0 + for field in template.fields: + # The length of the value byte slice is defined in the template + flen = field.field_length + if flen == 4: + struct_format += 'L' + elif flen == 2: + struct_format += 'H' + elif flen == 1: + struct_format += 'B' + else: + struct_format += f'{flen}s' + struct_len += flen + while offset <= (self.length - padding_size): - new_record = V9DataRecord() + unpacked_values = struct.unpack(struct_format, data[offset:offset + struct_len]) - for field in template.fields: - # The length of the value byte slice is defined in the template + new_record = V9DataRecord() + for field, value in zip(template.fields, unpacked_values): flen = field.field_length fkey = V9_FIELD_TYPES[field.field_type] # Special handling of IP addresses to convert integers to strings to not lose precision in dump # TODO: might only be needed for IPv6 if field.field_type in FIELD_TYPES_CONTAINING_IP: - dataslice = data[offset:offset+flen] try: - ip = ipaddress.ip_address(dataslice) + ip = ipaddress.ip_address(value) except ValueError: - print("IP address could not be parsed: {}".format(fdata)) + print("IP address could not be parsed: {}".format(repr(value))) continue new_record.data[fkey] = ip.compressed + elif flen in (1, 2, 4): + # These values are already converted to numbers by struct.unpack: + new_record.data[fkey] = value else: - # For performance reasons, we use struct.unpack for known lengths: - if flen == 4: - new_record.data[fkey], = struct.unpack_from('!L', data, offset) - elif flen == 2: - new_record.data[fkey], = struct.unpack_from('!H', data, offset) - elif flen == 1: - new_record.data[fkey], = struct.unpack_from('!B', data, offset) - else: - dataslice = data[offset:offset+flen] - # Caveat: this code assumes little-endian system (like x86) - fdata = 0 - for idx, byte in enumerate(reversed(bytearray(dataslice))): - fdata += byte << (idx * 8) - new_record.data[fkey] = fdata + # Caveat: this code assumes little-endian system (like x86) + fdata = 0 + for idx, byte in enumerate(reversed(bytearray(value))): + fdata += byte << (idx * 8) + new_record.data[fkey] = fdata offset += flen From 92b221aa105afe8cb581a29c108272223458c28d Mon Sep 17 00:00:00 2001 From: Anze Date: Sun, 8 May 2022 22:01:34 +0200 Subject: [PATCH 10/34] Fix: f-strings might not be supported --- netflow/v9.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/netflow/v9.py b/netflow/v9.py index 4723313..adfd169 100644 --- a/netflow/v9.py +++ b/netflow/v9.py @@ -222,7 +222,7 @@ def __init__(self, data, template): elif flen == 1: struct_format += 'B' else: - struct_format += f'{flen}s' + struct_format += '%ds' % flen struct_len += flen while offset <= (self.length - padding_size): From 7ea24a900cba696727e387c327b40eed2d136cf8 Mon Sep 17 00:00:00 2001 From: Dominik Pataky Date: Sat, 2 Jul 2022 11:50:18 +0200 Subject: [PATCH 11/34] Small addition to grafolean fix (comments, endianness hint) --- netflow/v9.py | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/netflow/v9.py b/netflow/v9.py index adfd169..aee63db 100644 --- a/netflow/v9.py +++ b/netflow/v9.py @@ -14,6 +14,7 @@ import ipaddress import struct +import sys from .ipfix import IPFIXFieldTypes, IPFIXDataTypes @@ -21,9 +22,7 @@ "V9TemplateFlowSet", "V9TemplateNotRecognized", "V9TemplateRecord", "V9OptionsTemplateFlowSet", "V9OptionsTemplateRecord", "V9OptionsDataRecord"] - -FIELD_TYPES_CONTAINING_IP = [8, 12, 27, 28] - +V9_FIELD_TYPES_CONTAINING_IP = [8, 12, 15, 18, 27, 28, 62, 63] V9_FIELD_TYPES = { 0: 'UNKNOWN_FIELD_TYPE', # fallback for unknown field types @@ -209,7 +208,8 @@ def __init__(self, data, template): padding_size = 4 - (self.length % 4) # 4 Byte # For performance reasons, we use struct.unpack to get individual values. Here - # we prepare the format string for parsing it: + # we prepare the format string for parsing it. The format string is based on the template fields and their + # lengths. The string can then be re-used for every data record in the data stream struct_format = '!' struct_len = 0 for field in template.fields: @@ -226,6 +226,8 @@ def __init__(self, data, template): struct_len += flen while offset <= (self.length - padding_size): + # Here we actually unpack the values, the struct format string is used in every data record + # iteration, until the final offset reaches the end of the whole data stream unpacked_values = struct.unpack(struct_format, data[offset:offset + struct_len]) new_record = V9DataRecord() @@ -235,7 +237,7 @@ def __init__(self, data, template): # Special handling of IP addresses to convert integers to strings to not lose precision in dump # TODO: might only be needed for IPv6 - if field.field_type in FIELD_TYPES_CONTAINING_IP: + if field.field_type in V9_FIELD_TYPES_CONTAINING_IP: try: ip = ipaddress.ip_address(value) except ValueError: @@ -247,6 +249,9 @@ def __init__(self, data, template): new_record.data[fkey] = value else: # Caveat: this code assumes little-endian system (like x86) + if sys.byteorder != "little": + print("v9.py uses bit shifting for little endianness. Your processor is not little endian") + fdata = 0 for idx, byte in enumerate(reversed(bytearray(value))): fdata += byte << (idx * 8) @@ -394,7 +399,7 @@ def __init__(self, data: bytes, template: V9OptionsTemplateRecord): for scope_type, length in template.scope_fields.items(): type_name = V9_SCOPE_TYPES.get(scope_type, scope_type) # Either name, or unknown int - value = int.from_bytes(data[offset:offset+length], 'big') # TODO: is this always integer? + value = int.from_bytes(data[offset:offset + length], 'big') # TODO: is this always integer? new_options_record.scopes[type_name] = value offset += length @@ -413,9 +418,9 @@ def __init__(self, data: bytes, template: V9OptionsTemplateRecord): value = None if is_bytes: - value = data[offset:offset+length] + value = data[offset:offset + length] else: - value = int.from_bytes(data[offset:offset+length], 'big') + value = int.from_bytes(data[offset:offset + length], 'big') new_options_record.data[type_name] = value From a94ad57f3e008d6ab9899640096666c421160b41 Mon Sep 17 00:00:00 2001 From: Dominik Pataky Date: Sat, 2 Jul 2022 12:09:29 +0200 Subject: [PATCH 12/34] IPFIX: fix usage of field data type (instead of name) Refs #40 --- netflow/ipfix.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/netflow/ipfix.py b/netflow/ipfix.py index c800f94..139100a 100644 --- a/netflow/ipfix.py +++ b/netflow/ipfix.py @@ -739,7 +739,7 @@ def __init__(self, data, template: List[Union[TemplateField, TemplateFieldEnterp raise NotImplementedError("Field type with ID {} is not implemented".format(field_type_id)) datatype = field_type.type # type: str - discovered_fields.append((field_type.name, field_type_id)) + discovered_fields.append((datatype, field_type_id)) # Catch fields which are meant to be raw bytes and skip the rest if IPFIXDataTypes.is_bytes(datatype): @@ -766,15 +766,15 @@ def __init__(self, data, template: List[Union[TemplateField, TemplateFieldEnterp pack = struct.unpack(unpacker, data[0:offset]) # Iterate through template again, but taking the unpacked values this time - for index, ((field_type_name, field_type_id), value) in enumerate(zip(discovered_fields, pack)): + for index, ((field_datatype, field_type_id), value) in enumerate(zip(discovered_fields, pack)): if type(value) is bytes: # Check if value is raw bytes, so no conversion happened in struct.unpack - if field_type_name in ["string"]: + if field_datatype in ["string"]: value = str(value) # TODO: handle octetArray (= does not have to be unicode encoded) - elif field_type_name in ["boolean"]: + elif field_datatype in ["boolean"]: value = True if value == 1 else False # 2 = false per RFC - elif field_type_name in ["dateTimeMicroseconds", "dateTimeNanoseconds"]: + elif field_datatype in ["dateTimeMicroseconds", "dateTimeNanoseconds"]: seconds = value[:4] fraction = value[4:] value = (int.from_bytes(seconds, "big"), int.from_bytes(fraction, "big")) From 4e065644fa850419bbc82948712b110dc5e1581f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Paul=20Gla=C3=9F?= Date: Fri, 2 Sep 2022 15:23:59 +0200 Subject: [PATCH 13/34] Fixed string conversion from bytes --- netflow/ipfix.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/netflow/ipfix.py b/netflow/ipfix.py index 139100a..f385b24 100644 --- a/netflow/ipfix.py +++ b/netflow/ipfix.py @@ -770,7 +770,7 @@ def __init__(self, data, template: List[Union[TemplateField, TemplateFieldEnterp if type(value) is bytes: # Check if value is raw bytes, so no conversion happened in struct.unpack if field_datatype in ["string"]: - value = str(value) + value = value.decode() # TODO: handle octetArray (= does not have to be unicode encoded) elif field_datatype in ["boolean"]: value = True if value == 1 else False # 2 = false per RFC From affef1a97266b914178234d88ab132641efad31a Mon Sep 17 00:00:00 2001 From: GitOldGrumpy Date: Fri, 2 Sep 2022 16:01:11 +0100 Subject: [PATCH 14/34] Fix for clearing enteprise flag bit --- netflow/ipfix.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/netflow/ipfix.py b/netflow/ipfix.py index 139100a..a0cb6e0 100644 --- a/netflow/ipfix.py +++ b/netflow/ipfix.py @@ -954,7 +954,7 @@ def parse_fields(data: bytes, count: int) -> (list, int): pack = struct.unpack("!HHI", data[offset:offset + 8]) fields.append( TemplateFieldEnterprise( - id=pack[0] & ~(1 << 7), # ID, clear enterprise flag bit + id=pack[0] & ~(1 << 15), # ID, clear enterprise flag bit length=pack[1], # field length enterprise_number=pack[2] # enterprise number ) From 0afa4b37e8f6276d570a838a29be18a53631065a Mon Sep 17 00:00:00 2001 From: Dominik Pataky Date: Fri, 2 Dec 2022 17:24:55 +0100 Subject: [PATCH 15/34] IPFIX: extend string field conversion to fallback to str() Closes #42 --- netflow/ipfix.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/netflow/ipfix.py b/netflow/ipfix.py index f385b24..3b21f9f 100644 --- a/netflow/ipfix.py +++ b/netflow/ipfix.py @@ -770,7 +770,10 @@ def __init__(self, data, template: List[Union[TemplateField, TemplateFieldEnterp if type(value) is bytes: # Check if value is raw bytes, so no conversion happened in struct.unpack if field_datatype in ["string"]: - value = value.decode() + try: + value = value.decode() + except UnicodeDecodeError: + value = str(value) # TODO: handle octetArray (= does not have to be unicode encoded) elif field_datatype in ["boolean"]: value = True if value == 1 else False # 2 = false per RFC From dbfc737e43185f5b0169a75120df0be73fbb847e Mon Sep 17 00:00:00 2001 From: Dominik Pataky Date: Sat, 3 Dec 2022 08:55:20 +0100 Subject: [PATCH 16/34] Fix test runner to ubuntu-20.04 --- .github/workflows/run_tests.yml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/run_tests.yml b/.github/workflows/run_tests.yml index 4687276..b5856c3 100644 --- a/.github/workflows/run_tests.yml +++ b/.github/workflows/run_tests.yml @@ -1,4 +1,4 @@ -name: Run tests +name: Run Python unit tests on: push: @@ -10,15 +10,15 @@ on: jobs: test: - runs-on: ubuntu-latest + runs-on: ubuntu-20.04 steps: # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it - uses: actions/checkout@v2 - + - name: Set up Python 3.5.3 uses: gabrielfalcao/pyenv-action@v7 with: default: '3.5.3' # Debian Stretch (oldoldstable) - + - name: Run Python unittests run: python3 -m unittest From 88f864036b8d989136f19014468feee53c10426c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Paul=20Gla=C3=9F?= Date: Fri, 2 Sep 2022 15:23:59 +0200 Subject: [PATCH 17/34] Fixed string conversion from bytes --- netflow/ipfix.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/netflow/ipfix.py b/netflow/ipfix.py index a0cb6e0..bb68069 100644 --- a/netflow/ipfix.py +++ b/netflow/ipfix.py @@ -770,7 +770,7 @@ def __init__(self, data, template: List[Union[TemplateField, TemplateFieldEnterp if type(value) is bytes: # Check if value is raw bytes, so no conversion happened in struct.unpack if field_datatype in ["string"]: - value = str(value) + value = value.decode() # TODO: handle octetArray (= does not have to be unicode encoded) elif field_datatype in ["boolean"]: value = True if value == 1 else False # 2 = false per RFC From 073a2122902a380e0c4ff484cf44b3f76170f9a4 Mon Sep 17 00:00:00 2001 From: Dominik Pataky Date: Fri, 2 Dec 2022 17:24:55 +0100 Subject: [PATCH 18/34] IPFIX: extend string field conversion to fallback to str() Closes #42 --- netflow/ipfix.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/netflow/ipfix.py b/netflow/ipfix.py index bb68069..d8eb412 100644 --- a/netflow/ipfix.py +++ b/netflow/ipfix.py @@ -770,7 +770,10 @@ def __init__(self, data, template: List[Union[TemplateField, TemplateFieldEnterp if type(value) is bytes: # Check if value is raw bytes, so no conversion happened in struct.unpack if field_datatype in ["string"]: - value = value.decode() + try: + value = value.decode() + except UnicodeDecodeError: + value = str(value) # TODO: handle octetArray (= does not have to be unicode encoded) elif field_datatype in ["boolean"]: value = True if value == 1 else False # 2 = false per RFC From 2eb2283873f7e3ae33f62a343f11a05a6f3cd2ac Mon Sep 17 00:00:00 2001 From: Dominik Pataky Date: Sat, 3 Dec 2022 08:55:20 +0100 Subject: [PATCH 19/34] Fix test runner to ubuntu-20.04 --- .github/workflows/run_tests.yml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/run_tests.yml b/.github/workflows/run_tests.yml index 4687276..b5856c3 100644 --- a/.github/workflows/run_tests.yml +++ b/.github/workflows/run_tests.yml @@ -1,4 +1,4 @@ -name: Run tests +name: Run Python unit tests on: push: @@ -10,15 +10,15 @@ on: jobs: test: - runs-on: ubuntu-latest + runs-on: ubuntu-20.04 steps: # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it - uses: actions/checkout@v2 - + - name: Set up Python 3.5.3 uses: gabrielfalcao/pyenv-action@v7 with: default: '3.5.3' # Debian Stretch (oldoldstable) - + - name: Run Python unittests run: python3 -m unittest From fe1d3df296be8f494debcfce85d769ad8a437f2e Mon Sep 17 00:00:00 2001 From: Dominik Pataky Date: Sat, 3 Dec 2022 09:33:24 +0100 Subject: [PATCH 20/34] IPFIX: improve bitwise operation on enterprise flag bit --- netflow/ipfix.py | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/netflow/ipfix.py b/netflow/ipfix.py index d8eb412..1524f5a 100644 --- a/netflow/ipfix.py +++ b/netflow/ipfix.py @@ -953,11 +953,11 @@ def parse_fields(data: bytes, count: int) -> (list, int): offset = 0 fields = [] # type: List[Union[TemplateField, TemplateFieldEnterprise]] for ctr in range(count): - if data[offset] & 1 << 7 != 0: # enterprise flag set + if (data[offset] & (1 << 7)) != 0: # enterprise flag set. Bitwise AND checks bit only in the first byte/octet pack = struct.unpack("!HHI", data[offset:offset + 8]) fields.append( TemplateFieldEnterprise( - id=pack[0] & ~(1 << 15), # ID, clear enterprise flag bit + id=(pack[0] & ~(1 << 15)), # clear enterprise flag bit. Bitwise AND and INVERT work on two bytes length=pack[1], # field length enterprise_number=pack[2] # enterprise number ) @@ -966,7 +966,10 @@ def parse_fields(data: bytes, count: int) -> (list, int): else: pack = struct.unpack("!HH", data[offset:offset + 4]) fields.append( - TemplateField(id=pack[0], length=pack[1]) + TemplateField( + id=pack[0], + length=pack[1] + ) ) offset += 4 return fields, offset From a410ef7de9c00ac1d86d426b1c76c94e14e0cac1 Mon Sep 17 00:00:00 2001 From: Dominik Pataky Date: Mon, 30 Jan 2023 13:30:53 +0100 Subject: [PATCH 21/34] Bump to v0.11.3 Includes fixes for enterprise bit flag and templatefield type check. --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 20c512d..b04d2e6 100644 --- a/setup.py +++ b/setup.py @@ -7,7 +7,7 @@ setup( name='netflow', - version='0.11.2', + version='0.11.3', description='NetFlow v1, v5, v9 and IPFIX tool suite implemented in Python 3', long_description=long_description, long_description_content_type='text/markdown', From bbc75ff0ca5dfef71433596a4d4ef8d717c6cf41 Mon Sep 17 00:00:00 2001 From: Dominik Pataky Date: Mon, 30 Jan 2023 13:56:48 +0100 Subject: [PATCH 22/34] Update Github test workflow with matrix --- .github/workflows/run_tests.yml | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/.github/workflows/run_tests.yml b/.github/workflows/run_tests.yml index b5856c3..6db0c43 100644 --- a/.github/workflows/run_tests.yml +++ b/.github/workflows/run_tests.yml @@ -4,21 +4,26 @@ on: push: branches: [ master ] pull_request: - branches: [ master ] workflow_dispatch: jobs: - test: + test-netflow: runs-on: ubuntu-20.04 + strategy: + matrix: + python: + - "3.5.3" # Debian Stretch + - "3.7.3" # Debian Buster + - "3.9.3" # Debian Bullseye + - "3.11.1" # Debian Bookworm steps: - # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it - - uses: actions/checkout@v2 + - uses: actions/checkout@v3 - - name: Set up Python 3.5.3 - uses: gabrielfalcao/pyenv-action@v7 + - name: Set up Python with pyenv + uses: gabrielfalcao/pyenv-action@v11 with: - default: '3.5.3' # Debian Stretch (oldoldstable) + default: "${{ matrix.python }}" - name: Run Python unittests run: python3 -m unittest From d9859e4dc20bd0fc071fd8d1f51727a557878727 Mon Sep 17 00:00:00 2001 From: Dominik Pataky Date: Mon, 30 Jan 2023 14:27:16 +0100 Subject: [PATCH 23/34] Fix workflow pyenv matrix Python 3.9 and 3.11 There's no 3.9.3, only 3.9.2, which is also the correct Debian version. 3.11.1 is available in pyenv 2.3.8, but 2.3.7 is currently in use by the Github action. So 3.11 is used, which uses the latest patch version --- .github/workflows/run_tests.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/run_tests.yml b/.github/workflows/run_tests.yml index 6db0c43..9fa1d5b 100644 --- a/.github/workflows/run_tests.yml +++ b/.github/workflows/run_tests.yml @@ -15,8 +15,8 @@ jobs: python: - "3.5.3" # Debian Stretch - "3.7.3" # Debian Buster - - "3.9.3" # Debian Bullseye - - "3.11.1" # Debian Bookworm + - "3.9.2" # Debian Bullseye + - "3.11" # Debian Bookworm uses 3.11.1, but it's in a newer pyenv release steps: - uses: actions/checkout@v3 From bb0ab89615d25fe0fe7853c7dfc92d9a42b72131 Mon Sep 17 00:00:00 2001 From: Dominik Pataky Date: Sat, 19 Aug 2023 09:24:49 +0200 Subject: [PATCH 24/34] Squashed commit of branch feature/ipfix-padding: commit 63abf52ec640a019f8c45c1208f0dfb585641781 Padding: add offset!=length check to reduce safety check calls Adds another check when parsing a set. The check "offset != self.header.length" allows to skip the padding checks if the offset is the same as the length, not calling rest_is_padding_zeroes and wasting CPU time. commit 8d1cf9cac12c45c0af70591b646d898ba5c923fc Finish IPFIX padding handling Tested implementation of IPFIX set padding handling. Uses TK-Khaw's proposed no_padding_last_offset calculation, extended as modulo calculation to match multiple data set records. Tests were conducted by capturing live traffic on a test machine with tcpdump, then this capture file was read in by softflowd 1.1.0, with the collector.py as the export target. The exported IPFIX (v10) packets were then using both no padding and padding, so that tests could be validated. Closes #34 Signed-off-by: Dominik Pataky commit 51ce4eaa268e4bda5be89e1d430477d12fc8a72c Fix and optimize padding calculation for IPFIX sets. Refs #34 commit 9d3c4135385ca9714b7631a0c5af46feb891a9fb Author: Khaw Teng Kang Date: Tue Jul 5 16:29:12 2022 +0800 Reverted changes to template_record, data_length is now computed using field length in template. Signed-off-by: Khaw Teng Kang commit 3c4f8e62892876d4a2d42288843890b97244df55 IPFIX: handle padding (zero bytes) in sets Adds a check to each IPFIX set ID branch, checking if the rest of the bytes in this set is padding/zeroes. Refs #34 Signed-off-by: Dominik Pataky --- netflow/collector.py | 6 ++-- netflow/ipfix.py | 79 ++++++++++++++++++++++++++++++++++++-------- 2 files changed, 69 insertions(+), 16 deletions(-) diff --git a/netflow/collector.py b/netflow/collector.py index 0cc9538..81c27fc 100644 --- a/netflow/collector.py +++ b/netflow/collector.py @@ -19,9 +19,9 @@ import time from collections import namedtuple -from .ipfix import IPFIXTemplateNotRecognized -from .utils import UnknownExportVersion, parse_packet -from .v9 import V9TemplateNotRecognized +from netflow.ipfix import IPFIXTemplateNotRecognized +from netflow.utils import UnknownExportVersion, parse_packet +from netflow.v9 import V9TemplateNotRecognized RawPacket = namedtuple('RawPacket', ['ts', 'client', 'data']) ParsedPacket = namedtuple('ParsedPacket', ['ts', 'client', 'export']) diff --git a/netflow/ipfix.py b/netflow/ipfix.py index 1524f5a..57276ce 100644 --- a/netflow/ipfix.py +++ b/netflow/ipfix.py @@ -635,6 +635,10 @@ class IPFIXTemplateNotRecognized(KeyError): pass +class PaddingCalculationError(Exception): + pass + + class IPFIXHeader: """The header of the IPFIX export packet """ @@ -663,9 +667,6 @@ def __init__(self, data): offset += offset_add if len(self.fields) != self.field_count: raise IPFIXMalformedRecord - - # TODO: if padding is needed, implement here - self._length = offset def get_length(self): @@ -697,8 +698,6 @@ def __init__(self, data): raise IPFIXMalformedRecord offset += offset_add - # TODO: if padding is needed, implement here - self._length = offset def get_length(self): @@ -812,17 +811,29 @@ def __init__(self, data: bytes, templates): self.records = [] self._templates = {} - offset = IPFIXSetHeader.size + offset = IPFIXSetHeader.size # fixed size + if self.header.set_id == 2: # template set while offset < self.header.length: # length of whole set template_record = IPFIXTemplateRecord(data[offset:]) self.records.append(template_record) if template_record.field_count == 0: + # Should not happen, since RFC says "one or more" self._templates[template_record.template_id] = None else: self._templates[template_record.template_id] = template_record.fields offset += template_record.get_length() + # If the rest of the data is deemed to be too small for another + # template record, check existence of padding + if ( + offset != self.header.length + and self.header.length - offset <= 16 # 16 is chosen as a guess + and rest_is_padding_zeroes(data[:self.header.length], offset) + ): + # Rest should be padding zeroes + break + elif self.header.set_id == 3: # options template while offset < self.header.length: optionstemplate_record = IPFIXOptionsTemplateRecord(data[offset:]) @@ -834,16 +845,47 @@ def __init__(self, data: bytes, templates): optionstemplate_record.scope_fields + optionstemplate_record.fields offset += optionstemplate_record.get_length() + # If the rest of the data is deemed to be too small for another + # options template record, check existence of padding + if ( + offset != self.header.length + and self.header.length - offset <= 16 # 16 is chosen as a guess + and rest_is_padding_zeroes(data[:self.header.length], offset) + ): + # Rest should be padding zeroes + break + elif self.header.set_id >= 256: # data set, set_id is template id - while offset < self.header.length: - template = templates.get( - self.header.set_id) # type: List[Union[TemplateField, TemplateFieldEnterprise]] - if not template: - raise IPFIXTemplateNotRecognized - data_record = IPFIXDataRecord(data[offset:], template) + # First, get the template behind the ID. Returns a list of fields or raises an exception + template_fields = templates.get( + self.header.set_id) # type: List[Union[TemplateField, TemplateFieldEnterprise]] + if not template_fields: + raise IPFIXTemplateNotRecognized + + # All template fields have a known length. Add them all together to get the length of the data set. + dataset_length = functools.reduce(lambda a, x: a + x.length, template_fields, 0) + + # This is the last possible offset value possible if there's no padding. + # If there is padding, this value marks the beginning of the padding. + # Two cases possible: + # 1. No padding: then (4 + x * dataset_length) == self.header.length + # 2. Padding: then (4 + x * dataset_length + p) == self.header.length, + # where p is the remaining length of padding zeroes. The modulo calculates p + no_padding_last_offset = self.header.length - ((self.header.length - IPFIXSetHeader.size) % dataset_length) + + while offset < no_padding_last_offset: + data_record = IPFIXDataRecord(data[offset:], template_fields) self.records.append(data_record) offset += data_record.get_length() - self._length = offset + + # Safety check + if ( + offset != self.header.length + and not rest_is_padding_zeroes(data[:self.header.length], offset) + ): + raise PaddingCalculationError + + self._length = self.header.length def get_length(self): return self._length @@ -973,3 +1015,14 @@ def parse_fields(data: bytes, count: int) -> (list, int): ) offset += 4 return fields, offset + + +def rest_is_padding_zeroes(data: bytes, offset: int) -> bool: + if offset <= len(data): + # padding zeros, so rest of bytes must be summed to 0 + if sum(data[offset:]) != 0: + return False + return True + + # If offset > len(data) there is an error + raise ValueError From 97c99f51b3a6ca31b45e57f0469659b0d3bbc895 Mon Sep 17 00:00:00 2001 From: Dominik Pataky Date: Sat, 19 Aug 2023 10:57:01 +0200 Subject: [PATCH 25/34] Fix type comparison from '==' to 'is' --- netflow/analyzer.py | 2 +- netflow/ipfix.py | 10 +++++----- netflow/utils.py | 12 ++++++------ 3 files changed, 12 insertions(+), 12 deletions(-) diff --git a/netflow/analyzer.py b/netflow/analyzer.py index 71338b7..7448b84 100644 --- a/netflow/analyzer.py +++ b/netflow/analyzer.py @@ -204,7 +204,7 @@ def total_packets(self): if args.match_host: try: match_host = ipaddress.ip_address(args.match_host) - except ValueError as ex: + except ValueError: exit("IP address '{}' is neither IPv4 nor IPv6".format(args.match_host)) # Using a file and using stdin differ in their further usage for gzip.open diff --git a/netflow/ipfix.py b/netflow/ipfix.py index 57276ce..b5121a2 100644 --- a/netflow/ipfix.py +++ b/netflow/ipfix.py @@ -514,9 +514,9 @@ def get_type_unpack(cls, key: Union[int, str]) -> Optional[DataType]: :return: """ item = None - if type(key) == int: + if type(key) is int: item = cls.by_id(key) - elif type(key) == str: + elif type(key) is str: item = cls.by_name(key) if not item: return None @@ -575,7 +575,7 @@ def is_signed(cls, dt: Union[DataType, str]) -> bool: :return: """ fields = ["signed8", "signed16", "signed32", "signed64"] - if type(dt) == DataType: + if type(dt) is DataType: return dt.type in fields return dt in fields @@ -587,7 +587,7 @@ def is_float(cls, dt: Union[DataType, str]) -> bool: :return: """ fields = ["float32", "float64"] - if type(dt) == DataType: + if type(dt) is DataType: return dt.type in fields return dt in fields @@ -601,7 +601,7 @@ def is_bytes(cls, dt: Union[DataType, str]) -> bool: fields = ["octetArray", "string", "macAddress", "ipv4Address", "ipv6Address", "dateTimeMicroseconds", "dateTimeNanoseconds"] - if type(dt) == DataType: + if type(dt) is DataType: return dt.type in fields return dt in fields diff --git a/netflow/utils.py b/netflow/utils.py index 395eb21..c4603d9 100644 --- a/netflow/utils.py +++ b/netflow/utils.py @@ -31,8 +31,8 @@ def get_export_version(data): return struct.unpack('!H', data[:2])[0] -def parse_packet(data: Union[str, bytes], templates: Dict = None) -> Union[V1ExportPacket, V5ExportPacket, - V9ExportPacket, IPFIXExportPacket]: +def parse_packet(data: Union[str, bytes], templates: Dict = None) \ + -> Union[V1ExportPacket, V5ExportPacket, V9ExportPacket, IPFIXExportPacket]: """ Parse an exported packet, either from string (hex) or from bytes. @@ -66,10 +66,10 @@ def parse_packet(data: Union[str, bytes], templates: Dict = None) -> Union[V1Exp :param templates: The templates dictionary with keys 'netflow' and 'ipfix' (created if not existing). :return: The parsed packet, or an exception. """ - if type(data) == str: + if type(data) is str: # hex dump as string data = bytes.fromhex(data) - elif type(data) == bytes: + elif type(data) is bytes: # check representation based on utf-8 decoding result try: # hex dump as bytes, but not hex @@ -83,8 +83,8 @@ def parse_packet(data: Union[str, bytes], templates: Dict = None) -> Union[V1Exp if version in [9, 10] and templates is None: raise ValueError("{} packet detected, but no templates dict was passed! For correct parsing of packets with " - "templates, create a 'templates' dict and pass it into the 'parse_packet' function.".format( - "NetFlow v9" if version == 9 else "IPFIX")) + "templates, create a 'templates' dict and pass it into the 'parse_packet' function." + .format("NetFlow v9" if version == 9 else "IPFIX")) if version == 1: return V1ExportPacket(data) From d26914234de1127a81babbe6e37541f9466b9077 Mon Sep 17 00:00:00 2001 From: Dominik Pataky Date: Sat, 19 Aug 2023 10:58:28 +0200 Subject: [PATCH 26/34] Github Workflow: run tests on release branch --- .github/workflows/run_tests.yml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/.github/workflows/run_tests.yml b/.github/workflows/run_tests.yml index 9fa1d5b..a2ed4f8 100644 --- a/.github/workflows/run_tests.yml +++ b/.github/workflows/run_tests.yml @@ -2,9 +2,8 @@ name: Run Python unit tests on: push: - branches: [ master ] + branches: [ master, release ] pull_request: - workflow_dispatch: jobs: From a829d428bfb806b322caa647d6238a30a041ff6b Mon Sep 17 00:00:00 2001 From: Dominik Pataky Date: Sat, 19 Aug 2023 14:33:12 +0200 Subject: [PATCH 27/34] Add test for IPFIX padding handling --- netflow/ipfix.py | 2 +- tests/lib.py | 13 +++++++++++++ tests/test_ipfix.py | 18 +++++++++++++++++- 3 files changed, 31 insertions(+), 2 deletions(-) diff --git a/netflow/ipfix.py b/netflow/ipfix.py index b5121a2..3744f55 100644 --- a/netflow/ipfix.py +++ b/netflow/ipfix.py @@ -1025,4 +1025,4 @@ def rest_is_padding_zeroes(data: bytes, offset: int) -> bool: return True # If offset > len(data) there is an error - raise ValueError + raise ValueError("netflow.ipfix.rest_is_padding_zeroes received a greater offset value than there is data") diff --git a/tests/lib.py b/tests/lib.py index 7db164a..7867132 100644 --- a/tests/lib.py +++ b/tests/lib.py @@ -310,3 +310,16 @@ def single_packet(pkts): "000000400000000100000000000000008800060000000000123456affefeaffeaffeaffe08000054" \ "fde66f14e0f196090000affeaffeaffe2a044e42020000000000000000000223e58bc8ede58be4d8" \ "00000140000000040000000000000000e54c01bb0602060000000000affeaffeaffe123456affefe" + +PACKET_IPFIX_PADDING = "000a01c064e0b1900000000200000000000200480400001000080004000c00040016000400150004" \ + "0001000400020004000a0004000e0004003d00010088000100070002000b00020004000100060001" \ + "003c000100050001000200400401000e00080004000c000400160004001500040001000400020004" \ + "000a0004000e0004003d0001008800010020000200040001003c0001000500010002004808000010" \ + "001b0010001c001000160004001500040001000400020004000a0004000e0004003d000100880001" \ + "00070002000b00020004000100060001003c000100050001000200400801000e001b0010001c0010" \ + "00160004001500040001000400020004000a0004000e0004003d000100880001008b000200040001" \ + "003c00010005000100030022010000060001008f000400a000080131000401320004013000020052" \ + "0010040100547f0000017f000001ffff07d0ffff0ff7000000fc0000000300000000000000000001" \ + "08000104007f0000017f000001ffff07d0ffff0ff7000000fc000000030000000000000000000100" \ + "0001040000000100002a0000b2da0000018a0db59d2e000000010000000000017465737463617074" \ + "7572655f73696e67" diff --git a/tests/test_ipfix.py b/tests/test_ipfix.py index b2bed1b..feac3b6 100644 --- a/tests/test_ipfix.py +++ b/tests/test_ipfix.py @@ -13,7 +13,7 @@ import unittest from tests.lib import send_recv_packets, PACKET_IPFIX_TEMPLATE, PACKET_IPFIX, PACKET_IPFIX_ETHER, \ - PACKET_IPFIX_TEMPLATE_ETHER + PACKET_IPFIX_TEMPLATE_ETHER, PACKET_IPFIX_PADDING class TestFlowExportIPFIX(unittest.TestCase): @@ -98,3 +98,19 @@ def test_ipfix_contents_ether(self): self.assertTrue(hasattr(flow, "postDestinationMacAddress")) self.assertEqual(flow.sourceMacAddress, 0x123456affefe) self.assertEqual(flow.postDestinationMacAddress, 0xaffeaffeaffe) + + def test_ipfix_padding(self): + """ + Checks successful parsing of export packets that contain padding zeroes in an IPFIX set. + The padding in the example data is in between the last two data sets, so the successful parsing of the last + data set indicates correct handling of padding zero bytes. + """ + pkts, _, _ = send_recv_packets([PACKET_IPFIX_PADDING]) + self.assertEqual(len(pkts), 1) + p = pkts[0] + + # Check for length of whole export + self.assertEqual(p.export.header.length, 448) + + # Check a specific value of the last flow in the export. Success means correct handling of padding in the set + self.assertEqual(p.export.flows[-1].meteringProcessId, 45786) From 942df30bf61f0d5611107ca2bcb5cd4181bdd7ca Mon Sep 17 00:00:00 2001 From: Dominik Pataky Date: Sat, 19 Aug 2023 19:16:02 +0200 Subject: [PATCH 28/34] Release v0.12.1 --- README.md | 2 +- netflow/ipfix.py | 6 +++--- setup.py | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 1a35cfd..838a2a1 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ Version 9 is the first NetFlow version using templates. Templates make dynamical ![Data flow diagram](nf-workflow.png) -Copyright 2016-2021 Dominik Pataky +Copyright 2016-2023 Dominik Pataky Licensed under MIT License. See LICENSE. diff --git a/netflow/ipfix.py b/netflow/ipfix.py index 3744f55..de462ed 100644 --- a/netflow/ipfix.py +++ b/netflow/ipfix.py @@ -967,15 +967,15 @@ def __init__(self, data: bytes, templates: Dict[int, list]): raise IPFIXMalformedPacket @property - def contains_new_templates(self): + def contains_new_templates(self) -> bool: return self._contains_new_templates @property - def flows(self): + def flows(self) -> list[IPFIXTemplateRecord | IPFIXOptionsTemplateRecord | IPFIXDataRecord]: return self._flows @property - def templates(self): + def templates(self) -> dict[int, list]: return self._templates def __repr__(self): diff --git a/setup.py b/setup.py index b04d2e6..2db8134 100644 --- a/setup.py +++ b/setup.py @@ -7,7 +7,7 @@ setup( name='netflow', - version='0.11.3', + version='0.12.1', description='NetFlow v1, v5, v9 and IPFIX tool suite implemented in Python 3', long_description=long_description, long_description_content_type='text/markdown', From 098acc1ae910739706ac5b6a4fb074a27b67304b Mon Sep 17 00:00:00 2001 From: Dominik Pataky Date: Sat, 19 Aug 2023 19:22:43 +0200 Subject: [PATCH 29/34] Fix tests by removing type hints --- netflow/ipfix.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/netflow/ipfix.py b/netflow/ipfix.py index de462ed..88c0334 100644 --- a/netflow/ipfix.py +++ b/netflow/ipfix.py @@ -971,11 +971,11 @@ def contains_new_templates(self) -> bool: return self._contains_new_templates @property - def flows(self) -> list[IPFIXTemplateRecord | IPFIXOptionsTemplateRecord | IPFIXDataRecord]: + def flows(self): return self._flows @property - def templates(self) -> dict[int, list]: + def templates(self): return self._templates def __repr__(self): From 71fb316a24357c2466fb996dffa82348a1e17b00 Mon Sep 17 00:00:00 2001 From: Dominik Pataky Date: Sun, 20 Aug 2023 14:44:44 +0200 Subject: [PATCH 30/34] Hotfix release v0.12.2 --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 2db8134..1490218 100644 --- a/setup.py +++ b/setup.py @@ -7,7 +7,7 @@ setup( name='netflow', - version='0.12.1', + version='0.12.2', description='NetFlow v1, v5, v9 and IPFIX tool suite implemented in Python 3', long_description=long_description, long_description_content_type='text/markdown', From c661ca711883645d2ef8713a060229f102ca8d32 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 27 Mar 2026 13:46:34 +0000 Subject: [PATCH 31/34] Initial plan From 662a5f4e034955dd1792b77bce6c5900ee882a3b Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 27 Mar 2026 13:49:09 +0000 Subject: [PATCH 32/34] Fix typos and step numbering errors in README.md documentation Agent-Logs-Url: https://github.com/neilalb16/python-netflow-v9-softflowd/sessions/cebd9b5e-c206-45e9-b68c-85513e654e33 Co-authored-by: neilalb16 <209672690+neilalb16@users.noreply.github.com> --- README.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 838a2a1..3593193 100644 --- a/README.md +++ b/README.md @@ -69,20 +69,20 @@ The test files contain tests for all use cases in the library, based on real sof 1. Run tcpdump/Wireshark on your public-facing interface (with tcpdump, save the pcap to disk). 2. Produce some sample flows, e.g. surf the web and refresh your mail client. With Wireshark, save the captured packets to disk. - 4. Run tcpdump/Wireshark again on a local interface. + 3. Run tcpdump/Wireshark again on a local interface. 4. Run `softflowd` with the `-r ` flag. softflowd reads the captured traffic, produces the flows and exports them. Use the interface you are capturing packets on to send the exports to. E.g. capture on the localhost interface (with `-i lo` or on loopback) and then let softflowd export to `127.0.0.1:1337`. 5. Examine the captured traffic. Use Wireshark and set the `CFLOW` "decode as" dissector on the export packets (e.g. based on the port). The `data` fields should then be shown correctly as Netflow payload. 6. Extract this payload as hex stream. Anonymize the IP addresses with a hex editor if necessary. A recommended hex editor is [bless](https://github.com/afrantzis/bless). Second, a Docker way: - 2. Run a softflowd daemon in the background inside a Docker container, listening on `eth0` and exporting to e.g. `172.17.0.1:1337`. - 3. On your host start Wireshark to listen on the Docker bridge. - 4. Create some traffic from inside the container. - 5. Check the softflow daemon with `softflowctl dump-flows`. - 6. If you have some flows shown to you, export them with `softflowctl expire-all`. - 7. Your Wireshark should have picked up the epxort packets (it does not matter if there's a port unreachable error). - 8. Set the decoder for the packets to `CFLOW` and copy the hex value from the NetFlow packet. + 1. Run a softflowd daemon in the background inside a Docker container, listening on `eth0` and exporting to e.g. `172.17.0.1:1337`. + 2. On your host start Wireshark to listen on the Docker bridge. + 3. Create some traffic from inside the container. + 4. Check the softflow daemon with `softflowctl dump-flows`. + 5. If you have some flows shown to you, export them with `softflowctl expire-all`. + 6. Your Wireshark should have picked up the export packets (it does not matter if there's a port unreachable error). + 7. Set the decoder for the packets to `CFLOW` and copy the hex value from the NetFlow packet. Your exported hex string should begin with `0001`, `0005`, `0009` or `000a`, depending on the version. From 9924f242cb06e8b3c66bf453de4c4580aab595a1 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 4 Apr 2026 17:07:24 +0000 Subject: [PATCH 33/34] Initial plan From 61a9e1da7ca1d2202496719bd20c8e55636b27ce Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 4 Apr 2026 17:45:19 +0000 Subject: [PATCH 34/34] Initial plan