Kernel dying with large max_distance value

[tlnagy]

I have ultrack running successfully on my machine, but I keep getting kernel dying errors during the Gurobi optimization section.

track(
    detection=detection,
    edges=edges,
    config=config,
    overwrite = True
)

Output from track function:

[/tmp/ipykernel_2094928/1028540271.py:1](http://localhost:8889/tmp/ipykernel_2094928/1028540271.py#line=0): DeprecationWarning: Argument detection is deprecated, please use foreground instead.
  track(
Adding nodes to database: 100%|| 1129[/1129](http://localhost:8889/1129) [01:24<00:00, 13.35it[/s](http://localhost:8889/s)]
Linking nodes.: 100%| 1128[/1128](http://localhost:8889/1128) [00:16<00:00, 67.79it[/s](http://localhost:8889/s)]
Set parameter Username
Set parameter LicenseID to value ~~~
Academic license - for non-commercial use only - expires 2025-11-25
Using Gurobi solver
Solving ILP batch 0
Constructing ILP ...
Set parameter TimeLimit to value 36000
Solving ILP ...
Set parameter NodeLimit to value 1073741824
Set parameter SolutionLimit to value 1073741824
Set parameter Threads to value 48
Set parameter IntFeasTol to value 1e-06
Set parameter Method to value 3
Set parameter MIPGap to value 0
Gurobi Optimizer version 11.0.3 build v11.0.3rc0 (linux64 - "Ubuntu 24.04.1 LTS")

CPU model: AMD Ryzen Threadripper 7960X 24-Cores, instruction set [SSE2|AVX|AVX2|AVX512]
Thread count: 24 physical cores, 48 logical processors, using up to 48 threads

Optimize a model with 188940 rows, 317031 columns and 618948 nonzeros
Model fingerprint: 0x4d328046
Variable types: 0 continuous, 317031 integer (317031 binary)
Coefficient statistics:
  Matrix range     [1e+00, 1e+00]
  Objective range  [2e-07, 1e+00]
  Bounds range     [1e+00, 1e+00]
  RHS range        [1e+00, 1e+00]
Found heuristic solution: objective -0.0000000
Presolve removed 113647 rows and 200251 columns (presolve time = 9s) ...
Presolve removed 116488 rows and 204451 columns
Presolve time: 9.91s
Presolved: 72452 rows, 112580 columns, 238871 nonzeros
Found heuristic solution: objective 407.7266821
Variable types: 0 continuous, 112580 integer (112580 binary)
Found heuristic solution: objective 407.9647214
Concurrent LP optimizer: primal simplex, dual simplex, and barrier
Showing barrier log only...

Root barrier log...

Ordering time: 0.03s

Barrier statistics:
 AA' NZ     : 1.594e+05
 Factor NZ  : 8.123e+05 (roughly 80 MB of memory)
 Factor Ops : 1.101e+07 (less than 1 second per iteration)
 Threads    : 46

                  Objective                Residual
Iter       Primal          Dual         Primal    Dual     Compl     Time
   0   2.79569520e+04  1.43765795e+05  1.45e+01 4.57e-01  2.81e+00    10s
   1  -2.92106923e+03  8.60588683e+04  2.05e+00 1.78e-15  5.40e-01    10s
   2  -2.36452095e+03  3.21694450e+04  1.34e-01 1.11e-15  1.33e-01    10s
   3  -8.18805802e+02  7.00766151e+03  5.89e-03 6.66e-16  2.86e-02    10s
   4   1.93871382e+02  3.89524898e+03  4.66e-04 5.55e-16  1.34e-02    10s
   5   4.50254612e+02  2.28085248e+03  1.89e-04 4.44e-16  6.65e-03    10s
   6   6.29126729e+02  1.82037826e+03  7.70e-05 4.44e-16  4.32e-03    10s
   7   7.19628383e+02  1.43488578e+03  3.89e-05 4.44e-16  2.60e-03    10s
   8   7.90760975e+02  1.22156421e+03  1.58e-05 4.44e-16  1.56e-03    10s
   9   8.27394835e+02  1.06449781e+03  6.78e-06 5.55e-16  8.61e-04    10s
  10   8.55613164e+02  9.51208823e+02  1.74e-06 5.55e-16  3.47e-04    11s

Barrier performed 10 iterations in 10.52 seconds (9.67 work units)
Barrier solve interrupted - model solved by another algorithm


Solved with dual simplex

Root simplex log...

Iteration    Objective       Primal Inf.    Dual Inf.      Time
   31636    8.7509205e+02   0.000000e+00   0.000000e+00     11s

Root relaxation: objective 8.750921e+02, 31636 iterations, 0.55 seconds (0.53 work units)

    Nodes    |    Current Node    |     Objective Bounds      |     Work
 Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

     0     0  875.09205    0 4693  407.96472  875.09205   115%     -   11s
H    0     0                     658.7062487  875.09205  32.9%     -   11s
H    0     0                     661.8973448  875.09205  32.2%     -   11s
H    0     0                     662.7405145  875.09205  32.0%     -   11s
H    0     0                     695.4382177  875.09205  25.8%     -   11s
     0     0  850.71011    0 3913  695.43822  850.71011  22.3%     -   12s
H    0     0                     739.0322215  850.71011  15.1%     -   12s
     0     0  837.64598    0 3793  739.03222  837.64598  13.3%     -   12s
H    0     0                     756.0684541  837.50382  10.8%     -   12s
     0     0  837.50382    0 3855  756.06845  837.50382  10.8%     -   12s
     0     0  837.50382    0 3856  756.06845  837.50382  10.8%     -   12s
     0     0  830.79218    0 3858  756.06845  830.79218  9.88%     -   13s
H    0     0                     767.7948548  830.79218  8.20%     -   13s
     0     0  830.64825    0 4090  767.79485  830.64825  8.19%     -   13s
     0     0  830.64775    0 4109  767.79485  830.64775  8.19%     -   13s
     0     0  828.20101    0 3978  767.79485  828.20101  7.87%     -   13s
     0     0  828.13005    0 4089  767.79485  828.13005  7.86%     -   13s
     0     0  828.11916    0 4016  767.79485  828.11916  7.86%     -   14s
     0     0  828.11500    0 4022  767.79485  828.11500  7.86%     -   14s
     0     0  826.14471    0 4073  767.79485  826.14471  7.60%     -   14s
H    0     0                     772.8419371  826.14471  6.90%     -   14s
     0     0  826.08114    0 4236  772.84194  826.08114  6.89%     -   14s
     0     0  826.08000    0 4237  772.84194  826.08000  6.89%     -   14s
     0     0  824.42198    0 4087  772.84194  824.42198  6.67%     -   14s
     0     0  824.34368    0 4176  772.84194  824.34368  6.66%     -   15s
     0     0  824.34300    0 4172  772.84194  824.34300  6.66%     -   15s
     0     0  823.43267    0 4185  772.84194  823.43267  6.55%     -   15s
H    0     0                     774.8492450  823.43267  6.27%     -   15s
     0     0  823.38701    0 4303  774.84924  823.38701  6.26%     -   15s
     0     0  823.37265    0 4319  774.84924  823.37265  6.26%     -   15s
     0     0  823.36074    0 4325  774.84924  823.36074  6.26%     -   15s
     0     0  822.58761    0 3960  774.84924  822.58761  6.16%     -   15s
     0     0  822.50336    0 3973  774.84924  822.50336  6.15%     -   16s
     0     0  822.47825    0 3989  774.84924  822.47825  6.15%     -   16s
     0     0  822.46208    0 3986  774.84924  822.46208  6.14%     -   16s
     0     0  821.72674    0 3926  774.84924  821.72674  6.05%     -   16s
H    0     0                     776.8357455  821.72674  5.78%     -   16s
     0     0  821.71443    0 3949  776.83575  821.71443  5.78%     -   16s
     0     0  821.08470    0 3808  776.83575  821.08470  5.70%     -   16s
H    0     0                     776.8357460  821.08470  5.70%     -   17s
     0     0  821.06777    0 3849  776.83575  821.06777  5.69%     -   17s
     0     0  821.06773    0 3851  776.83575  821.06773  5.69%     -   17s
     0     0  820.76810    0 3829  776.83575  820.76810  5.66%     -   17s
H    0     0                     778.5181653  820.76810  5.43%     -   17s
     0     0  820.74354    0 3847  778.51817  820.74354  5.42%     -   17s
     0     0  820.74326    0 3853  778.51817  820.74326  5.42%     -   17s
     0     0  820.44235    0 3927  778.51817  820.44235  5.39%     -   17s
     0     0  820.43314    0 3910  778.51817  820.43314  5.38%     -   18s
     0     0  820.11744    0 3960  778.51817  820.11744  5.34%     -   18s
H    0     0                     779.1750137  820.11744  5.25%     -   18s
     0     0  820.10800    0 3919  779.17501  820.10800  5.25%     -   18s
     0     0  820.03471    0 3899  779.17501  820.03471  5.24%     -   18s
     0     0  820.01683    0 3934  779.17501  820.01683  5.24%     -   19s
     0     0  820.01298    0 3959  779.17501  820.01298  5.24%     -   19s
     0     0  819.92805    0 4052  779.17501  819.92805  5.23%     -   19s
     0     0  819.92190    0 4036  779.17501  819.92190  5.23%     -   19s
H    0     0                     779.3067816  819.81546  5.20%     -   20s
H    0     0                     779.3067820  819.81546  5.20%     -   28s
H    0     0                     780.6325894  819.81546  5.02%     -   30s
H    0     0                     781.5093424  819.81546  4.90%     -   33s
     0     2  819.81546    0 4036  781.50934  819.81546  4.90%     -   34s
    31    64  819.81546    5 3864  781.50934  819.81546  4.90%  25.7   35s
H   36    64                     781.5093425  819.81546  4.90%  22.8   35s
H   38    64                     781.5268444  819.81546  4.90%  21.9   35s
H  111   160                     784.8423494  819.81546  4.46%  17.1   40s
H  112   160                     784.8423498  819.81546  4.46%  17.3   40s
H  117   160                     784.8423503  819.81546  4.46%  18.5   40s
  1210  1259  819.14368   19 3590  784.84235  819.81546  4.46%  10.3   45s
H 1211  1259                     784.8556275  819.81546  4.45%  10.2   45s
H 1256  1259                     784.8562549  819.81546  4.45%  10.2   45s
H 1257  1259                     785.3637259  819.81546  4.39%  10.2   45s
H 2330  2379                     787.3338414  819.81546  4.13%   8.1   95s
H 2331  2379                     787.8680812  819.81546  4.05%   8.1   95s
H 2332  2379                     788.5483790  819.81546  3.97%   8.1   95s
H 2335  2379                     788.8018050  819.81546  3.93%   8.1   95s
H 2377  2379                     788.8518404  819.81546  3.93%   8.0   95s
H 2378  2427                     788.8679493  819.81546  3.92%   8.0   99s
H 2380  2427                     788.8679499  819.81546  3.92%   8.0   99s
H 2397  2427                     789.3882545  819.81546  3.85%   8.0   99s
H 2408  2427                     789.3882547  819.81546  3.85%   7.9   99s
  2571  2734  817.27501   35 3404  789.38825  819.81546  3.85%   7.9  100s
H 3423  3467                     789.4262907  819.81546  3.85%   7.3  103s
H 3430  3467                     789.4262916  819.81546  3.85%   7.3  103s
  3693  3912  816.02947   48 3377  789.42629  819.81546  3.85%   7.1  105s
H 4687  4733                     789.4317502  819.81546  3.85%   6.9  108s
  4999  5230  814.13173   71 3300  789.43175  819.81546  3.85%   7.0  110s
  5918  6218  813.26707   84 3162  789.43175  819.81546  3.85%   6.8  115s
H 7732  8099                     789.7528860  819.81546  3.81%   6.8  120s
H 7968  8099                     790.1676212  819.81546  3.75%   6.8  121s
  9061  9257  809.60959  135 2639  790.16762  819.81546  3.75%   6.8  125s
H 9742  9777                     790.4692550  819.81546  3.71%   6.8  131s
H 9744  9777                     791.1385952  819.81546  3.62%   6.8  131s
H 9745  9777                     791.1385954  819.81546  3.62%   6.8  131s
H 9750  9777                     791.1385957  819.81546  3.62%   6.8  131s
 10818 11039  808.68674  166 2410  791.13860  819.81546  3.62%   6.7  136s
 11479 12090  807.56343  177 2370  791.13860  819.81546  3.62%   6.8  140s
 13489 14063  804.99291  208 2178  791.13860  819.81546  3.62%   6.8  146s
 16539 17682  801.18807  252 1933  791.13860  819.81546  3.62%   6.8  151s
H17963 16797                     791.3079931  819.81546  3.60%   6.8  380s
H17968 15961                     791.9014070  800.21303  1.05%   6.8  381s
H17979 15170                     791.9014073  797.16650  0.66%   6.8  381s
H17990 14417                     792.4168600  796.64810  0.53%   6.8  383s
H18000 13702                     792.4168603  796.45272  0.51%   6.8  385s
H18018 13028                     792.4168605  796.29218  0.49%   6.8  388s
 18028 13035  796.21872   14 1212  792.41686  796.21872  0.48%   6.8  390s
 18055 13053  796.12807   45 1352  792.41686  796.12807  0.47%   6.8  395s
 18088 13076  796.01179  178 1306  792.41686  796.01179  0.45%   8.5  400s
H18095 12425                     792.4168609  795.98304  0.45%   8.5  401s
 18116 12439  795.81053  198 1407  792.41686  795.81053  0.43%   8.5  405s
 18148 12461  795.61721  195 1394  792.41686  795.61721  0.40%   8.5  410s
 18453 12713  795.50412   32 1286  792.41686  795.56003  0.40%  11.2  415s
 19914 13702  794.89548   46  941  792.41686  795.56003  0.40%  13.1  420s
 23052 15813  793.98248   81  573  792.41686  795.56003  0.40%  18.3  425s
 27488 17923  793.47734  122  457  792.41686  795.56003  0.40%  20.8  430s
 31973 19686  792.84581  167  316  792.41686  795.56003  0.40%  22.3  450s
H31994 19054                     792.4168616  795.56003  0.40%  22.3  450s
 34699 21150  793.61955   55  868  792.41686  795.53967  0.39%  23.4  455s
 41124 24205  793.15832   81  579  792.41686  795.51431  0.39%  24.9  460s
 47421 27417  792.94162  121  449  792.41686  795.51078  0.39%  25.5  465s
 54424 31510  794.34189   38  991  792.41686  795.41063  0.38%  26.4  470s
 61330 36089  793.54377   73  607  792.41686  795.39499  0.38%  27.3  476s
 66954 42776  793.85470   55  962  792.41686  795.34980  0.37%  27.5  480s
 70724 45571  794.15920   39  961  792.41686  795.34883  0.37%  27.5  485s
 73608 45572  794.17266   46 4036  792.41686  795.34883  0.37%  27.8  509s
 73614 45576  792.67489   66 1223  792.41686  795.34883  0.37%  27.7  510s
 73637 45591  794.16802   51 1263  792.41686  795.32363  0.37%  27.7  515s
H73641 43313                     792.4168620  795.26212  0.36%  27.7  517s
 73651 43320  794.30107   47 1323  792.41686  795.19253  0.35%  27.7  520s
 73664 43328  793.40290   76 1398  792.41686  795.13562  0.34%  27.7  525s
 73677 43337  794.14392   52 1421  792.41686  795.07852  0.34%  27.7  530s
 73691 43346  793.06672   61 1495  792.41686  795.04883  0.33%  27.7  535s
 73696 43350  794.34987   46 1469  792.41686  795.04500  0.33%  27.7  540s
 73705 43356  794.36500   42 1450  792.41686  795.04080  0.33%  27.7  545s
 73714 43362  792.67489   66 1524  792.41686  795.03886  0.33%  27.7  550s
 73722 43367  792.57730  110 1522  792.41686  795.03755  0.33%  27.7  555s
 73728 43371  792.67827   61 1526  792.41686  795.02017  0.33%  27.7  564s
 73729 43375  795.02017   38 1496  792.41686  795.02017  0.33%  28.2  565s
 74298 43832  794.13831   48 1170  792.41686  794.93892  0.32%  28.6  570s
 75970 44702  793.07176   64  703  792.41686  794.93892  0.32%  29.2  575s
 79724 46431  794.48002   57 1154  792.41686  794.93794  0.32%  29.3  580s
 84133 48899  794.25218   63 1129  792.41686  794.91838  0.32%  29.5  585s
 92183 52973  793.50675   75  939  792.41686  794.68421  0.29%  29.7  590s
 100078 56009  792.75917   77  735  792.41686  794.58099  0.27%  29.8  595s
 106294 59896  793.52446   62 1007  792.41686  794.47272  0.26%  30.1  600s
 114002 63603  794.02082   50 1061  792.41686  794.41215  0.25%  30.2  605s
 122344 67822  792.96613  100  611  792.41686  794.37990  0.25%  30.5  610s
 129684 71177  793.15458   83  560  792.41686  794.36260  0.25%  31.0  615s
 135628 71652  792.96554  110  398  792.41686  794.35856  0.25%  31.1  622s
 138650 75673  793.39046   67  861  792.41686  794.33674  0.24%  31.2  625s
 147098 80963  793.24324   77  548  792.41686  794.31143  0.24%  31.3  633s
 152662 82338  792.59850  102  368  792.41686  794.29974  0.24%  31.3  635s
 161117 86125  794.13203   46 1004  792.41686  794.28406  0.24%  31.3  640s
 170035 88760  793.72366   70 1019  792.41686  794.26785  0.23%  31.5  646s
 178442 95400  793.07889   88  474  792.41686  794.25007  0.23%  31.6  653s
 184495 99292  793.45978   68  743  792.41686  794.24460  0.23%  31.4  657s
 192421 101316  793.01134   91  583  792.41686  794.23294  0.23%  31.4  660s
 204064 105649  793.10569   70  946  792.41686  794.21223  0.23%  31.6  668s
 209906 110849  793.23877   70  903  792.41686  794.20508  0.23%  31.8  671s
 216381 115388  793.61070   61  757  792.41686  794.19502  0.22%  31.8  675s
 222689 115442  793.14838   81  686  792.41686  794.18789  0.22%  31.9  694s
 222754 134743  793.12321   82  703  792.41686  794.18755  0.22%  31.9  713s
 248194 137140     cutoff   73       792.41686  794.15279  0.22%  32.3  719s
 252285 147166  793.52869   60  959  792.41686  794.14739  0.22%  32.4  726s
 265134 152349  793.57178   56  941  792.41686  794.13446  0.22%  32.5  732s
 274091 157940  793.91908   54  922  792.41686  794.12163  0.22%  32.9  737s
 281731 163165  793.02418   83  816  792.41686  794.11949  0.21%  32.9  741s
 294523 173298  793.30845   74  989  792.41686  794.09926  0.21%  33.0  749s
 302847 180723  792.91800   82  707  792.41686  794.08852  0.21%  33.0  755s
 320486 193037  792.96272   70  693  792.41686  794.07160  0.21%  33.0  764s
 328483 198039  792.87654   95  437  792.41686  794.06537  0.21%  32.9  768s
 334952 203800  793.49625   62  928  792.41686  794.06067  0.21%  33.0  773s
 342951 211644  793.07053   76 1014  792.41686  794.05571  0.21%  33.0  778s
 353086 216370  792.97285   82  899  792.41686  794.04771  0.21%  33.0  783s
 360028 218725  793.31846   65  908  792.41686  794.04344  0.21%  33.1  787s
 363286 224307  793.10712   82  814  792.41686  794.04056  0.20%  33.1  792s
 372028 227795  792.56006   74  485  792.41686  794.03509  0.20%  33.3  797s
 378071 233400  792.65270   55  780  792.41686  794.03234  0.20%  33.5  803s
 387067 239037  792.72167   85  547  792.41686  794.02818  0.20%  33.7  809s
 395909 245058  793.15608   62  936  792.41686  794.02387  0.20%  33.9  814s
 404230 250808  792.50736   77  677  792.41686  794.02011  0.20%  33.9  818s
 411922 256982  793.01061   84  672  792.41686  794.01675  0.20%  33.9  823s
 419870 262679  793.13362   81  893  792.41686  794.01285  0.20%  33.8  827s
 427463 269676  792.45970   76  783  792.41686  794.00870  0.20%  33.8  832s
 437152 274295  792.59934   81   95  792.41686  794.00393  0.20%  33.8  836s
 444186 279282  792.78457   56  677  792.41686  794.00151  0.20%  33.8  841s
 451456 284274  792.51226   66  512  792.41686  793.99748  0.20%  33.9  845s
 458652 289547     cutoff  106       792.41686  793.99394  0.20%  34.0  850s
 473396 297974  793.22144   75  800  792.41686  793.98876  0.20%  34.1  859s
 477951 306170  793.78109   71  866  792.41686  793.98686  0.20%  34.1  865s
 488873 312531  792.59520   68  281  792.41686  793.98240  0.20%  34.1  870s
 497901 318151  792.63622   60  570  792.41686  793.97928  0.20%  34.2  875s
 505967 324879  792.95674   65  906  792.41686  793.97593  0.20%  34.3  880s
 516258 330534  792.67555   74  934  792.41686  793.97181  0.20%  34.2  885s
 529975 340911  792.43498   68  785  792.41686  793.96694  0.20%  34.3  893s
 537200 346452  792.83135   73  766  792.41686  793.96490  0.20%  34.3  898s
 544788 352761  792.52895   71  621  792.41686  793.96172  0.19%  34.3  903s
 553596 357250  792.57557   72  720  792.41686  793.95905  0.19%  34.3  907s
 559639 363393  793.15487   76  960  792.41686  793.95683  0.19%  34.3  911s
 567947 368500  793.33595   69  863  792.41686  793.95406  0.19%  34.3  916s
 575666 374442  792.49427   73  306  792.41686  793.95203  0.19%  34.4  921s
 583448 380344  792.45700   91  671  792.41686  793.95023  0.19%  34.4  926s
 591704 385899     cutoff   73       792.41686  793.94765  0.19%  34.4  931s
 599901 388653  793.49363   58  935  792.41686  793.94534  0.19%  34.4  935s
 603866 395561  792.41866   97  299  792.41686  793.94416  0.19%  34.4  940s
 612784 400257  793.04530   73  652  792.41686  793.94171  0.19%  34.5  945s
 620412 406893  792.52972   75  576  792.41686  793.94041  0.19%  34.6  950s
 636664 416040  793.52686   59  894  792.41686  793.93583  0.19%  34.6  958s
 641466 421622  792.87473   76  697  792.41686  793.93492  0.19%  34.6  962s
 649482 427180  792.99223   72  863  792.41686  793.93264  0.19%  34.6  967s
 657788 432545  792.56022   68  485  792.41686  793.93072  0.19%  34.6  973s
 665783 437753  793.03037   64  786  792.41686  793.92913  0.19%  34.7  978s
 673816 443548  792.86896   69  808  792.41686  793.92704  0.19%  34.8  982s
 681370 448755  792.67814   70  717  792.41686  793.92506  0.19%  34.7  987s
 689125 453949  792.43519   68  356  792.41686  793.92329  0.19%  34.8  991s
 696550 460122  792.74088   79  664  792.41686  793.92138  0.19%  34.8  996s
 705085 465550  792.75172   62  761  792.41686  793.91953  0.19%  34.8 1001s
 712343 470804  792.63001   84  452  792.41686  793.91753  0.19%  34.8 1005s
 719677 475765  793.41126   58  893  792.41686  793.91627  0.19%  34.8 1010s
 730283 481540     cutoff   96       792.41686  793.91415  0.19%  34.9 1020s
~~~ snip ~~~
 5808854 3628071     cutoff   80       792.41686  793.49999  0.14%  37.8 4317s
 5816387 3632997  792.82712   79  330  792.41686  793.49973  0.14%  37.8 4323s
 5824049 3637352  792.90101   90  443  792.41686  793.49948  0.14%  37.8 4327s
 5831212 3641233  793.45437   58  715  792.41686  793.49925  0.14%  37.8 4331s
 5837314 3644605  792.47173  112   93  792.41686  793.49921  0.14%  37.8 4337s
 5845330 3649510  792.62278   72  648  792.41686  793.49885  0.14%  37.8 4342s
 5852982 3653843     cutoff  100       792.41686  793.49866  0.14%  37.8 4347s
 5860401 3658418  792.60204   67  863  792.41686  793.49835  0.14%  37.8 4352s
 5868066 3663088  792.62379   86  239  792.41686  793.49823  0.14%  37.8 4357s
 5875786 3667291  793.41426   58  967  792.41686  793.49784  0.14%  37.8 4362s
 5882711 3671905  792.67842  102  241  792.41686  793.49768  0.14%  37.8 4366s
 5889641 3676755  792.90363   65  826  792.41686  793.49748  0.14%  37.8 4371s
 5897206 3681798  793.19297   58  826  792.41686  793.49716  0.14%  37.8 4376s
 5904706 3686869  792.88033   68  934  792.41686  793.49691  0.14%  37.8 4381s
 5912424 3690316  792.90120   66  812  792.41686  793.49666  0.14%  37.8 4385s
 5918302 3694883  793.04808   60  670  792.41686  793.49649  0.14%  37.8 4390s
 5925901 3699734  792.94579   70  565  792.41686  793.49624  0.14%  37.8 4395s
 5933196 3704395  793.16652   69  798  792.41686  793.49596  0.14%  37.8 4400s
 5946793 3713264  792.87147   71  677  792.41686  793.49551  0.14%  37.8 4409s
 5955274 3717936  792.42151   68  288  792.41686  793.49523  0.14%  37.8 4414s
 5963257 3722045  792.43235   81  742  792.41686  793.49497  0.14%  37.8 4420s
 5978287 3731291  792.52913   81  413  792.41686  793.49451  0.14%  37.8 4429s
 5985847 3734008  792.45841   76  209  792.41686  793.49427  0.14%  37.8 4433s
 5991837 3737865  793.30886   73  913  792.41686  793.49404  0.14%  37.8 4438s
 5999911 3741350  792.66209   83  757  792.41686  793.49388  0.14%  37.8 4444s
 6007316 3745436     cutoff  123       792.41686  793.49371  0.14%  37.9 4449s
 6014758 3750018  793.11956   66  749  792.41686  793.49346  0.14%  37.8 4454s
 6021887 3754534  792.88817   78  734  792.41686  793.49322  0.14%  37.8 4459s
 6029818 3759525  792.83442   68  700  792.41686  793.49294  0.14%  37.9 4463s
 6037540 3764608     cutoff   64       792.41686  793.49271  0.14%  37.8 4468s
 6045330 3768560  792.83430   75  643  792.41686  793.49253  0.14%  37.8 4473s
 6052423 3772414  792.54309   92  715  792.41686  793.49231  0.14%  37.9 4478s
 6059643 3776318  792.60310  106  222  792.41686  793.49214  0.14%  37.9 4482s
 6065799 3780776  792.75573   63  784  792.41686  793.49188  0.14%  37.9 4487s
 6073765 3784565     cutoff   84       792.41686  793.49163  0.14%  37.9 4492s
 6081051 3788666     cutoff   89       792.41686  793.49152  0.14%  37.9 4497s
 6088358 3793168  793.18907   65  675  792.41686  793.49125  0.14%  37.9 4502s
 6095961 3797371  792.56756   78  780  792.41686  793.49110  0.14%  37.9 4508s
 6103434 3802041  792.69937   79  599  792.41686  793.49084  0.14%  37.9 4513s
 6111077 3805763  792.56871   73  591  792.41686  793.49065  0.14%  37.9 4518s
 6117595 3811624  792.69425   67  306  792.41686  793.49043  0.14%  37.9 4523s
 6126296 3816478  792.50659   68  538  792.41686  793.49015  0.14%  37.9 4528s
 6134146 3821513  793.02337   74  498  792.41686  793.48985  0.14%  37.9 4533s
 6142097 3823589     cutoff   84       792.41686  793.48973  0.14%  37.9 4537s
 6145382 3827913  793.06142   73  657  792.41686  793.48964  0.14%  37.9 4542s
 6153022 3832384  793.09287   75  535  792.41686  793.48937  0.14%  37.9 4547s
 6160701 3837363  792.72567   78  625  792.41686  793.48918  0.14%  37.9 4552s
 6168240 3841579  792.72163  119  267  792.41686  793.48891  0.14%  37.9 4556s
 6175573 3845813  793.24258   72  882  792.41686  793.48870  0.14%  37.9 4561s
 6183053 3850858  792.51461   80  459  792.41686  793.48851  0.14%  37.9 4566s
 6190843 3855514  792.51228   86  610  792.41686  793.48828  0.14%  37.9 4570s
 6198145 3859716  792.65944   66  735  792.41686  793.48803  0.14%  37.9 4575s
 6205832 3863446  792.51887   83  503  792.41686  793.48780  0.14%  37.9 4580s
 6211830 3868244  792.83822   59  910  792.41686  793.48767  0.14%  37.9 4585s
 6219948 3872929     cutoff   88       792.41686  793.48736  0.14%  37.9 4590s
 6235353 3881943  792.88587   71  827  792.41686  793.48690  0.14%  37.9 4599s
 6242257 3886352  792.52322  106  111  792.41686  793.48674  0.14%  37.9 4604s
 6249773 3890078  792.44038   70  313  792.41686  793.48650  0.13%  37.9 4610s
 6257313 3895078  792.52457   74  591  792.41686  793.48638  0.13%  37.9 4615s
 6272138 3904033  792.49352   70  492  792.41686  793.48587  0.13%  37.9 4625s
 6282050 3910925  792.60369   67  368  792.41686  793.48556  0.13%  37.9 4635s
 6292168 3916015  792.45966   67  489  792.41686  793.48531  0.13%  37.9 4641s
 6301445 3920724  792.54189   71  595  792.41686  793.48493  0.13%  37.9 4646s
 6308886 3925097  792.48570   66  707  792.41686  793.48474  0.13%  37.9 4651s
 6316561 3929802     cutoff   90       792.41686  793.48450  0.13%  37.9 4655s
 6323986 3934961     cutoff   76       792.41686  793.48429  0.13%  37.9 4660s
 6331663 3940136  792.87495   68  781  792.41686  793.48412  0.13%  37.9 4665s
 6339287 3944146     cutoff   67       792.41686  793.48386  0.13%  37.9 4670s
 6346612 3948192  792.69227   68  452  792.41686  793.48370  0.13%  37.9 4675s
 6354274 3952191  792.88218   64  793  792.41686  793.48349  0.13%  37.9 4680s
 6367544 3960950  792.57535  107  219  792.41686  793.48311  0.13%  37.9 4689s
 6375869 3965869  793.03756   73  875  792.41686  793.48283  0.13%  37.9 4694s
 6383545 3970005  793.03784   70  705  792.41686  793.48261  0.13%  37.9 4699s
 6390784 3973747  792.85691   75  848  792.41686  793.48242  0.13%  37.9 4704s
 6398121 3977776  792.89422   70  462  792.41686  793.48219  0.13%  37.9 4709s
 6405630 3982011  792.51944   76  503  792.41686  793.48197  0.13%  37.9 4714s
 6412991 3986749  793.03099   72  563  792.41686  793.48171  0.13%  37.9 4719s
 6420624 3990798  792.82527   70  761  792.41686  793.48147  0.13%  37.9 4724s
 6427962 3994380  792.52924   98  337  792.41686  793.48133  0.13%  37.9 4729s
 6435103 3999079  792.91027   64  748  792.41686  793.48104  0.13%  37.9 4734s
 6442881 4002884  793.00261   61  716  792.41686  793.48082  0.13%  37.9 4738s
 6449028 4006683  792.64611   56  705  792.41686  793.48060  0.13%  37.9 4743s
 6456277 4010247     cutoff   83       792.41686  793.48035  0.13%  37.9 4748s
 6464068 4014315  792.57994   92  220  792.41686  793.48019  0.13%  37.9 4753s
 6471638 4018932  792.43759   65  687  792.41686  793.47994  0.13%  37.9 4758s
 6479168 4022970  792.85787   64  793  792.41686  793.47983  0.13%  37.9 4763s
 6486517 4027018     cutoff   90       792.41686  793.47955  0.13%  38.0 4768s
 6494020 4031591  792.46597   72  507  792.41686  793.47930  0.13%  38.0 4773s
 6501405 4035835  792.59218   90  220  792.41686  793.47909  0.13%  38.0 4777s
 6509070 4039823  792.54312   69  591  792.41686  793.47884  0.13%  38.0 4782s
 6516508 4043872  792.91960   71  731  792.41686  793.47863  0.13%  38.0 4787s
 6523717 4047503  792.44730   80  553  792.41686  793.47845  0.13%  38.0 4792s
 6529960 4053565  792.59465   75  610  792.41686  793.47826  0.13%  38.0 4798s
 6539021 4058661  792.65296   61  763  792.41686  793.47797  0.13%  38.0 4803s
 6546894 4062823  792.99688   62  785  792.41686  793.47771  0.13%  38.0 4808s
 6554697 4067762     cutoff   74       792.41686  793.47746  0.13%  38.0 4813s
 6562494 4072781  792.98511   75  624  792.41686  793.47724  0.13%  38.0 4818s
 6570222 4077015  793.05682   66  620  792.41686  793.47702  0.13%  38.0 4823s
 6577423 4081003     cutoff   78       792.41686  793.47679  0.13%  38.0 4827s
 6584824 4085177  792.71952  102  333  792.41686  793.47656  0.13%  38.0 4832s
 6592160 4088881     cutoff   66       792.41686  793.47638  0.13%  38.0 4837s
 6598595 4093133  792.63573  100  220  792.41686  793.47622  0.13%  38.0 4842s
 6606373 4097379  792.62814   73  405  792.41686  793.47596  0.13%  38.0 4847s
 6613657 4100459  792.45155  122  218  792.41686  793.47578  0.13%  38.0 4851s
 6620703 4105266  793.07037   60 1012  792.41686  793.47557  0.13%  38.0 4856s
 6628401 4109790  792.42509   81  651  792.41686  793.47538  0.13%  38.0 4861s
 6636056 4113617  792.90894   66  625  792.41686  793.47514  0.13%  38.0 4866s
 6643350 4118132  792.89916   69  818  792.41686  793.47485  0.13%  38.0 4871s
 6651345 4122205  792.57060   84  313  792.41686  793.47459  0.13%  38.0 4876s
 6658824 4124553  792.54308   79  596  792.41686  793.47437  0.13%  38.0 4880s
 6664304 4130064     cutoff   78       792.41686  793.47421  0.13%  38.0 4885s
 6673051 4134094  793.04691   60  758  792.41686  793.47398  0.13%  38.0 4891s
 6680041 4137906  792.77414   64  675  792.41686  793.47380  0.13%  38.0 4896s
 6687883 4143082 infeasible   71       792.41686  793.47351  0.13%  38.0 4901s
 6695756 4146779  792.43568   78  273  792.41686  793.47329  0.13%  38.0 4905s
 6701748 4151478  792.87340   79  425  792.41686  793.47309  0.13%  38.0 4911s
 6710118 4156037  792.51512   65  539  792.41686  793.47283  0.13%  38.0 4916s
 6717691 4160813  792.58188   74  480  792.41686  793.47265  0.13%  38.0 4920s
 6725107 4165428  793.00115   68  676  792.41686  793.47234  0.13%  38.0 4925s
 6732510 4169372  792.77965   76  905  792.41686  793.47211  0.13%  38.0 4930s
 6741937 4175289  792.96861   72  707  792.41686  793.47196  0.13%  38.0 4940s
 6750923 4180036  792.49779   83  641  792.41686  793.47159  0.13%  38.0 4945s
 6758508 4184167  792.92361   76  438  792.41686  793.47139  0.13%  38.0 4950s
 6765638 4188818  792.43850   73  360  792.41686  793.47119  0.13%  38.0 4955s
 6773333 4193483  792.83434   78  724  792.41686  793.47101  0.13%  38.0 4960s
 6780806 4197384  792.91873   80  780  792.41686  793.47078  0.13%  38.0 4965s
 6788102 4202126  792.86615   68  761  792.41686  793.47057  0.13%  38.0 4970s
 6795425 4204067  792.64335   81  596  792.41686  793.47033  0.13%  38.0 4975s
 6799215 4209902  792.84765   65  804  792.41686  793.47020  0.13%  38.0 4982s
 6809273 4215990  792.93845   60  767  792.41686  793.46998  0.13%  38.0 4988s
 6819484 4221216     cutoff   70       792.41686  793.46963  0.13%  38.0 4994s
 6828233 4225858     cutoff   72       792.41686  793.46935  0.13%  38.0 4999s
 6835660 4230490  792.57222   74  683  792.41686  793.46920  0.13%  38.0 5003s
 6843320 4234977  792.66608   67  591  792.41686  793.46891  0.13%  38.0 5008s
 6851038 4239016  792.43471   59  384  792.41686  793.46862  0.13%  38.0 5013s
 6858381 4243867  792.86445   75  672  792.41686  793.46837  0.13%  38.0 5018s
 6866008 4248534     cutoff   91       792.41686  793.46815  0.13%  38.0 5023s
 6873622 4253007  792.94009   74  704  792.41686  793.46790  0.13%  38.0 5027s
 6881008 4256488  792.84928   61  520  792.41686  793.46765  0.13%  38.0 5032s
 6887226 4261690  792.61245   70  636  792.41686  793.46750  0.13%  38.0 5037s
 6895567 4266131  792.53263  107  254  792.41686  793.46720  0.13%  38.0 5042s
 6903294 4270097  792.61834   73  858  792.41686  793.46698  0.13%  38.0 5047s
 6910510 4274688  792.50600   84  347  792.41686  793.46677  0.13%  38.0 5052s
 6918278 4278853  792.67694   77  484  792.41686  793.46650  0.13%  38.0 5057s
 6925945 4283019  792.88480   64  771  792.41686  793.46638  0.13%  38.1 5062s
 6933243 4287088  792.60504   67  619  792.41686  793.46613  0.13%  38.1 5067s
 6941063 4291675  792.61239   58  525  792.41686  793.46597  0.13%  38.1 5072s
 6948828 4295392  792.48253   91   28  792.41686  793.46568  0.13%  38.1 5077s
 6956109 4298561  792.46553   85  389  792.41686  793.46545  0.13%  38.1 5081s
 6961952 4303286  792.47890   80  686  792.41686  793.46532  0.13%  38.1 5086s
 6970223 4307854  792.91542   79  541  792.41686  793.46506  0.13%  38.1 5091s
 6978012 4312519  792.85922   71  810  792.41686  793.46484  0.13%  38.1 5096s
 6985569 4316652  792.51842   82  309  792.41686  793.46458  0.13%  38.1 5101s
 6993139 4321105  793.20809   58  867  792.41686  793.46435  0.13%  38.1 5106s
 7000900 4325779  792.79974  118  239  792.41686  793.46423  0.13%  38.1 5111s
 7008393 4330622  792.79355   86  398  792.41686  793.46398  0.13%  38.1 5115s
 7015937 4335368     cutoff   75       792.41686  793.46380  0.13%  38.1 5120s
 7029493 4342619  792.64645   64  726  792.41686  793.46341  0.13%  38.1 5130s
 7037614 4347303  792.52184   68  796  792.41686  793.46323  0.13%  38.1 5135s
 7045592 4350712     cutoff   73       792.41686  793.46302  0.13%  38.1 5140s
 7053035 4354495  792.80968   78  492  792.41686  793.46272  0.13%  38.1 5146s
 7060660 4359162  792.59694   86  285  792.41686  793.46262  0.13%  38.1 5151s
 7068260 4363517  793.02856   76  505  792.41686  793.46236  0.13%  38.1 5156s
 7075594 4368327  792.42860   67  790  792.41686  793.46219  0.13%  38.1 5161s
 7083351 4372073     cutoff   61       792.41686  793.46196  0.13%  38.1 5166s
 7090909 4376302  792.45209  106  266  792.41686  793.46175  0.13%  38.1 5171s
 7098197 4380751  792.66849   85  569  792.41686  793.46152  0.13%  38.1 5177s
 7106003 4384395  792.51417   89  359  792.41686  793.46126  0.13%  38.1 5181s
 7112541 4388973     cutoff   85       792.41686  793.46117  0.13%  38.1 5187s
 7120637 4393459  792.86310   70  892  792.41686  793.46083  0.13%  38.1 5193s

Config:

config.segmentation_config.min_area = 250
config.segmentation_config.max_area = 3000
config.segmentation_config.n_workers = 48

config.linking_config.max_distance = 50
config.linking_config.n_workers = 48

config.tracking_config.appear_weight = -1
config.tracking_config.disappear_weight = -0.1
config.tracking_config.division_weight = -0.1
config.tracking_config.power = 4
config.tracking_config.bias = -0.001
config.tracking_config.solution_gap = 0.0
config.tracking_config.window_size = 100
config.tracking_config.overlap_size = 5
config.tracking_config.n_threads = 48

My movie is 1129 frames of 3000x3000 images. It has about ~450 cells per frame with a big drop in cell number towards the end. Do you all have any suggestions on how to diagnose the kernel dying event? The kernel doesn't die with a lower max_distance, but the cells aren't properly linked at the start of the movie when they are moving quickly.

[JoOkuma]

Hi @tlnagy ,

Do you think you could be running out of memory?

Your problem is taking a long time to converge, so I would change the solution_gap to 0.005, the algorithm might be using more memory storing intermediate solutions to reach a global optimal.

These last few percents of the gap might change a handful associations and are often not critical.

I also suggest checking if you really benefit from the high number of parallel threads during tracking (config.tracking_config.n_threads).
Each thread in gurobi explores an instances in parallel, so you memory usage scale with the number of threads.
ILPs are a bit weird, sometimes more threads aren't very beneficial.

[tlnagy]

I have a 128GB of memory on my workstation and usage was hovering around 60% when I checked but I didn't track it carefully. I'll try with your suggestions and report back. Thanks!