feat: market #301
Draft
feat: market #301
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* webui: Jetbrains Mono, fix porep layout (#179) * Move storiface into Curio (#183) * temp: replace lotus to local * move storiface to Curio * fix build * don't refer to storiface2 * make gen * make remote path test happy * cleanup storiface * Batch deadline setting in SectorMetadata (#177) * basic mk12 scafolding * config comments, http header init * fix gen test * fix tests * poller redesign * remove indexing table, add defaults * fix ingest, market tasks * incomplete basic UI code * finish libp2p init, fix tests * fix test-all * enable libp2p relay * add url add command * fix circleCI * fix everything else * apply suggestion from review, cleanup todos * remove import-data * unify libp2p, atomic market scheduler, config changes * minor sql fixes * undo makefile changes * fix final review --------- Co-authored-by: Łukasz Magiera <[email protected]>
* basic mk12 scafolding * fix gen test * poller redesign * remove indexing table, add defaults * fix ingest, market tasks * incomplete basic UI code * finish libp2p init, fix tests * fix test-all * ipni provider * fix storeiface, publisher multiaddrs * close rows * apply suggestions * fix error * fix contextID type, move ipni in market * gen, remove old market sql
* create http server * remove loadbalancer, add db cache, set defaults * update config * fix lint err
* enables retrievals * fix itest * correct test path * apply changes from review
* wip simpler ipni * initial smart ipni chunker * ipni: Complete smart chunker * ipni: Use batching for initial chunker * ipni: wire up the better chunker * make gen * uncached ipni entry server * harmonydb: Add retry to .Exec * ipni: Address review * indexstore: Don't trim multihashes
* fix ipni http path, add cached reader * cachedPieceReader library * add getAdBytes method * close reader
* fix DealPublishControl address usage * fix forest test
* remove MarketAddBalance * make gen
* libp2p migration and generation * check libp2p rows * row check in market cli
* cuhttp: Fix startup issues * cachedreader: Don't reuse io.Reader * dealdata: Fix URLPieceReader * retrieval: Fix Routes * make gen * ipni: Fix construction * ipni: More column name fixes * fix libp2p key serialization * ipni: task logic fixes * ipni: get the task into working shape * make gen
* ipni: Fixed endpoints, working head * ipni: Fix Ad cids * ipni: fix chunk output, addrs in ad addresses * make gen * I think we need this * Or not
* libp2p: Fix startup, cleanup config * ipni: Separate key table * libp2p: Working single Libp2p * cuhttp: Libp2p redirector * make gen * libp2p: Automatically set actor addresses
* webui: Ask UI * webui market: Prices in tib/mo * webui market: Fix price calc * webui market: Working ask setting * fix libp2p safehandle * more fixes to get deals working * fix market navlink * last fix to create_indexing_task
* controlled indexing after migration * shared max counter, ipni complete * limit count
* deal list page, deal detail page * fix SQL typos
* feat: ipni ui * make gen --------- Co-authored-by: Łukasz Magiera <[email protected]>
* Add MoveEscrow to UI * fix gen
* add backpressure to mk12 deals * fix backpressure CTE
* rearrange pieces in sector for no padding * replace transfer funcs on go side * remove stateful piece offset
* deal filters * replace price rule number with name
magik6k
reviewed
Nov 6, 2024
* market: Improve indexing logging * Add missing ipni peerid constraint * cuhttp: allow external tls termination * better errors in indexing * don't error on nil errors * indexing: pass correct errors * indexing: More aggresive retry * indexing: More conservative batch size * ipni: log index rate * indexing: Schedule ipni first * indexing: Bump cql params * indexing: ab test sorted inserts * nope * indexing: Less consistency * ipni: Fix prev parse in serve * ipni: Set ad addresses when serving * ipni: Bump some log levels * ipni: debugging chunk reader * Export dagstore metrics * remotebstore: Fix metrics * export metrics properly this time * tune min read size * indexing: Try a completely different schema * use bufio for faster indexing readers * debug get piecehashrange * more ipni debug * ipni from-car also needs next node * check numblocks in indexstore * flag for ipni chunk debug * indexing: Fix offset checks * fastparamfetch: Improve aria opts * http is bad * ipni: Announce correct address * make gen * ipni/dealfilter fixes * more dealfilter fixes * fix http header marshal * fix max concurrent deal size backpressure * fix null error on deal status check * fix dealstatus/sector insert * fix panic in sector alloc * make gen * fix seal start duration * deal ui fixes * improve piece state webui * fix how indexing tasks are created * webui: move task page under pages * webui: task run page * webui: make sector page load fast * webui: Better snap page * webui: Add snap data to the sector page * webui: make piece page readable * webui: show multiple deals on the piece page * webui: Show related pipelines in task page * webui: nicer deal list * market: Fix sector to deal assignment * webui: Fix deal page pefore publish * webui: Improve deal status display * market: Fix sector offset finding * webui: show piecepark on the piece page * urlpiecereader: Improve errors * webui: Advanced task display on piece page * webui: Small deal page improvements * webui: Fix deal page on sealed deals * market: Correctly mark deals as sealed * indexing: Fix non-imported deal indexing * ipni: Limit max from-car entry size * ipni: Small entry cache to support retry after read timeout * ipni: Bump write timeout * ipni: Speculatively precompute next entry * ipni: More informative logs * market: Set deal sector regprooftype in ingest * make gen * fix snap deal ingest * webui: Show failed snap info on the sector page * park piece: better retry backoff * webui: Improve the Storage Deals page * Reindexing * fix ipni retrievals * cleanup indexing tasks more correctly * ipni debug * ipni: Detailed ad output, entry scan * webui: better scanned entry select * More ipni debugging * libp2p: Handle boost transports proto * make gen * http: Add missing info endpoint * ipni: Make db reads really work this time * storage: Show reservations in expvar * storage debug; proposal cid in deals * webui: Show proposal CID * webui: Nicer time format on tasks pages * ipni: wip always index from db * ipni: Auto-repair missing Ads * webui: Fix IPNI status * webui: typo in ipni * fix bronkn storage release * webui: Improve the piece page more * webui: Message Cid component * webui: Epoch component * ipni: fix bad entry panic * ipni: search by entry cid * ipni: Ad skip option * webui: Pipeline remove button * webui: pieceref lookup on the piece page * gen, mod tidy * snap ingest: fix sector size check * storage: Fix max storage pct math * webui: Fix ipni status page on fresh providers * ipni: Space provider announcments * webui: Pipeline waterfalls * webui: market waterfall * webui: sdr/snap pipeline waterfalls * market-specific backpressure knobs * cli: Drop testDebugMigPcid * webui: market bulk fail management * tasks: Put sealing before piecepark * webui: market bulk fail remove * gen, don't hardcode f0 * fix indexstore test * Add missing retry in movestorage * Appease the linter * fix indexstore test * make gen * Fix indexstore test for real * fix localpath test
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR replaces Boost with New Curio market module. Once this PR is merged, Boost will no longer be compatible with Curio.