You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Also, I believe the client shouldn't manage the schema directly anyway.
write with a higher concurrency
write with a larger batch
It might make sense to integrate this into the client. The idea would be that when the client receives a large batch of documents, it starts sending them to Manticore at the maximum possible speed, automatically adjusting concurrency and batch size. This could be based on factors like the number of CPU cores and workers available in the Manticore instance, increasing until further gains are no longer feasible.
changed the title [-]"The Manticore PHP client doesn't do the things recommended above out of the box."[/-][+]Method to upload a large batch at max speed[/+]on Sep 24, 2024
For ease of use, I copy-pasted part of the chat from the screenshot above:
"The PHP client doesn't do the things recommended above out of the box. If you want to measure max throughput on your server you can use a script like this (modify it, so the table schema/data looks like yours):"
#!/usr/bin/php
<?php
if (count($argv) < 4) die("Usage: ".__FILE__." <batch size> <concurrency> <docs>\n");
// This function waits for an idle mysql connection for the $query, runs it and exits
function process($query) {
global $all_links;
global $requests;
global $latencies;
foreach ($all_links as $k=>$link) {
if (@$requests[$k]) continue;
mysqli_query($link, $query, MYSQLI_ASYNC);
@$requests[$k] = microtime(true);
return true;
}
do {
$links = $errors = $reject = array();
foreach ($all_links as $link) {
$links[] = $errors[] = $reject[] = $link;
}
$count = @mysqli_poll($links, $errors, $reject, 0, 1000);
if ($count > 0) {
foreach ($links as $j=>$link) {
$res = @mysqli_reap_async_query($links[$j]);
foreach ($all_links as $i=>$link_orig) if ($all_links[$i] === $links[$j]) break;
if ($link->error) {
echo "ERROR in '".substr($query, 0, 100)."...': {$link->error}\n";
if (!mysqli_ping($link)) {
echo "ERROR: mysql connection is down, removing it from the pool\n";
unset($all_links[$i]); // remove the original link from the pool
unset($requests[$i]); // and from the $requests too
}
return false;
}
if ($res === false and !$link->error) continue;
if (is_object($res)) {
mysqli_free_result($res);
}
$latencies[] = microtime(true) - $requests[$i];
$requests[$i] = microtime(true);
mysqli_query($link, $query, MYSQLI_ASYNC); // making next query
return true;
}
};
} while (true);
return true;
}
$t = microtime(true);
$all_links = [];
$requests = [];
$latencies = [];
$c = 0;
for ($i=0;$i<$argv[2];$i++) {
$m = @mysqli_connect('127.0.0.1', '', '', '', 9315);
if (mysqli_connect_error()) die("Cannot connect to Manticore\n");
$all_links[] = $m;
// mysqli_query($all_links[0], "drop table if exists user$i");
// mysqli_query($all_links[0], "create table user$i(area text, age int, active bit(1)) hitless_words='all' rt_mem_limit='16M'");
}
// init
mysqli_query($all_links[0], "drop table if exists user");
mysqli_query($all_links[0], "create table user(area text, age int, active bit(1)) hitless_words='all' rt_mem_limit='16M'");
$batch = [];
//$query_start = "insert into user".rand(0,$argv[2])."(id, area, age, active) values ";
$query_start = "insert into user(id, area, age, active) values ";
$error = false;
while (count($all_links) and $c < $argv[3]) {
$batch[] = "(0,'".substr(md5(rand()), 0, 6)."',".rand(5,15).",".rand(0,1).")";
$c++;
if (count($batch) == $argv[1]) {
if (!process($query_start.implode(',', $batch))) {
$error = true;
break;
}
$batch = [];
}
}
// wait until all the workers finish
do {
$links = $errors = $reject = array();
foreach ($all_links as $link) $links[] = $errors[] = $reject[] = $link;
$count = @mysqli_poll($links, $errors, $reject, 0, 100);
} while (count($all_links) != count($links) + count($errors) + count($reject));
echo "finished inserting\n";
echo round($argv[3] / (microtime(true) - $t))." docs per sec\n";
Activity
sanikolaev commentedon Sep 24, 2024
Hi @marclaporte
This has been mitigated here manticoresoftware/manticoresearch#879
This will be resolved once auto-sharding is completed:
Also, I believe the client shouldn't manage the schema directly anyway.
It might make sense to integrate this into the client. The idea would be that when the client receives a large batch of documents, it starts sending them to Manticore at the maximum possible speed, automatically adjusting concurrency and batch size. This could be based on factors like the number of CPU cores and workers available in the Manticore instance, increasing until further gains are no longer feasible.
[-]"The Manticore PHP client doesn't do the things recommended above out of the box."[/-][+]Method to upload a large batch at max speed[/+]marclaporte commentedon Feb 19, 2025
Many things have been addressed.
One remaining blocker: manticoresoftware/manticoresearch#3073
marclaporte commentedon Mar 21, 2025
AFAICT, the last blocker is manticoresoftware/manticoresearch#1087
marclaporte commentedon Jun 9, 2026
Now that manticoresoftware/manticoresearch#1087 and manticoresoftware/manticoresearch#3073 have been handled, what is the next step, if any?
marclaporte commentedon Jun 14, 2026
This is most interesting! This could help us compare various configurations and optimize resource allocation.
marclaporte commentedon Jun 14, 2026
For ease of use, I copy-pasted part of the chat from the screenshot above:
"The PHP client doesn't do the things recommended above out of the box. If you want to measure max throughput on your server you can use a script like this (modify it, so the table schema/data looks like yours):"
sanikolaev commentedon Jun 15, 2026
The next step is to finally release the sharding functionality, which should happen soon. Once that's done we'll prioritize this task.
marclaporte commentedon Jul 2, 2026
Congratulations for auto-sharding and the multiple other enhancements in the latest release!
https://github.com/manticoresoftware/manticoresearch/releases/tag/27.1.5
marclaporte commentedon Aug 3, 2026
Sharding in Manticore Search: automatic distribution and replication
Author: Dmitrii Kuzmenkov
Published: Jul 02, 2026
https://manticoresearch.com/blog/sharding-in-manticore-search/#benchmarks-does-sharding-actually-speed-up-inserts
Nick-S-2018 commentedon Aug 27, 2026
Since we’ll soon introduce indexer support for RT tables, it makes sense to use that mode for bulk uploads in the PHP client as well.
https://github.com/manticoresoftware/manticoresearch-php/tree/rt-indexer-bulk
In our tests, it was roughly 2–4× faster than regular batched uploads, depending on the worker count.
Blocked by manticoresoftware/manticoresearch#4824
sanikolaev commentedon Sep 29, 2026
Unblocked.
marclaporte commentedon Oct 9, 2026
WIP: #271
Nick-S-2018 commentedon Nov 4, 2026
Implemented in #271
marclaporte commentedon Nov 5, 2026
I am super thrilled!