[onert] batchmatmul does not allow constant for lhs

[glistening]

I got error during running decode.circle which is a part of tinyllama.

I got the error:

$ ./Product/x86_64-linux.debug/out/bin/onert_run tinyllama/decode/decode.circle 
Model Filename tinyllama/decode/decode.circle
Fail to load model: OperationValidator failed at line 122

void OperationValidator::visit(const operation::BatchMatMul &node)
{
  ...
  // RHS can be constant, but LHS is not constant
  // If one of inputs is constant, it must be RHS
  // If two inputs are constant, BatchMatMul is optimized into constant by compiler
  OP_REQUIRES(!isConstant(lhs_index));

says onert does not allow const in lhs.

 

I generated decode.circle using TICO.
In addition, the same error occurs in other LLAMA variations (e.g. gemma, llama, ...).

We need to do:

• Allow constant lhs in onert

OR

• Add a pass in TICO not to generated this pattern

cc @seockho-kim

[glistening]

I tried to write code that uses fullyconnected internally for this case.
However, it requires more changes than I expected.
It would be better to handle this in Front End. (either TICO or circle2circle).

TICO official solution is proceeding in Samsung/TICO#339.

I also prepared the modified circle. See Samsung/TICO#339 (comment).

I will use this model for a while.

I hope the official TICO solution eliminate two transposes that does nothing as whole.

[glistening]

@seockho-kim tole me that just commenting out works.

onert is ready for batchmatmul lhs const.

It is just not allowed, I guess, for forcing fullyconnected which has efficient kernel.

void OperationValidator::visit(const operation::BatchMatMul &node)
{
  ...
  // RHS can be constant, but LHS is not constant
  // If one of inputs is constant, it must be RHS
  // If two inputs are constant, BatchMatMul is optimized into constant by compiler
  // OP_REQUIRES(!isConstant(lhs_index));

[glistening]

I succeed to make lhs-const-bmm-free circle model.

• Run TICO using [serialize] prefer fc over bmm on const lhs TICO#356.
• Run modify_fc.py, which modifies fc's 3d weight to 2d. AI assistant wrote it.
• Run circle2circle to remove redundant 2 transpose after fc.

Note

You need to specify --remove_redundant_transpose .
circle2circle does not remove redundant transposes by default. 🤔

$ circle2circle --remove_redundant_transpose {in,out}.circle

 

We can get fullyconnected using either #356 or forthcoming #359.