There is a memory leak, the model has dynamic input and dynamic output.

[ZzhangYyan]

Request Description

Application: Lightglue model for image registration, the input is a fixed-size image, the output is a dynamic shape. Openvino has a memory leak, I hope to solve this problem

Model source：https://github.com/fabio-sim/LightGlue-ONNX/releases/tag/v1.0.0

Openvino version is 2024.3.0.

code:
std::vector ApplyTransform1(const cv::Mat& image, float& mean, float& std)
{
cv::Mat resized, floatImage;
image.convertTo(floatImage, CV_32FC1);
mean = 0.0f;
std = 0.0f;
cv::Scalar meanScalar, stdScalar;
meanStdDev(floatImage, meanScalar, stdScalar);
mean = static_cast(meanScalar.val[0]);
std = static_cast(stdScalar.val[0]);

std::vector imgData;
for (int h = 0; h < image.rows; h++)
{
for (int w = 0; w < image.cols; w++)
{
imgData.push_back((floatImage.at(h, w) - mean) / std);
}
}
return imgData;
}

void MatchEnd2EndInf::SetImgInTensor(const cv::Mat& img, ov::Tensor& tensor)
{
std::vector imgData;
cv::Mat grayImg;
float mean, std;

if (img.channels() == 3)
{
cv::cvtColor(img, grayImg, cv::COLOR_BGR2GRAY);
}
else
{
/* grayImg.copyTo(img);*/
grayImg = img.clone();
}

imgData = ApplyTransform1(grayImg, mean, std);

float* data = tensor.data();
for (size_t i = 0; i < imgData.size(); i++)
{
data[i] = imgData[i];
}
}

void MatchEnd2EndInf::Infer(const cv::Mat& baseImg, const cv::Mat& testImg, cv::detail::MatchesInfo& matches_info)
{
std::chrono::steady_clock::time_point startTime;
std::chrono::steady_clock::time_point endTime;
std::chrono::duration<double, std::milli> duration;
int nElapsedTime;
int hh = 0, ww = 0;
ov::Shape shape1 = ov::Shape(4);
ov::Shape shape2 = ov::Shape(4);

hh = baseImg.rows; ww = baseImg.cols;
shape1[0] = 1;
shape1[1] = 1;
shape1[2] = hh;
shape1[3] = ww;

hh = testImg.rows; ww = testImg.cols;
shape2[0] = 1;
shape2[1] = 1;
shape2[2] = hh;
shape2[3] = ww;

startTime = std::chrono::steady_clock::now();
{
// -------- Step 3. Create an Inference Request --------
std::unique_ptrov::InferRequest inferRequest = std::make_uniqueov::InferRequest(this->m_pModel->create_infer_request());
ov::Tensor baseTensor(ov::element::f32, shape1);
ov::Tensor testTensor(ov::element::f32, shape2);

SetImgInTensor(baseImg, baseTensor);
SetImgInTensor(testImg, testTensor);

inferRequest->set_tensor("image0", baseTensor);
inferRequest->set_tensor("image1", testTensor);

std::cout << "The shape of output image0:" << baseTensor.get_shape() << std::endl;
std::cout << "The shape of output image1:" << testTensor.get_shape() << std::endl;

inferRequest->infer();

ov::Tensor output = inferRequest->get_output_tensor(0);
ov::Shape output_shape = output.get_shape();
ov::Tensor output1 = inferRequest->get_output_tensor(1);
ov::Shape output_shape1 = output1.get_shape();
ov::Tensor output2 = inferRequest->get_output_tensor(2);
ov::Shape output_shape2 = output2.get_shape();
ov::Tensor output3 = inferRequest->get_output_tensor(3);
ov::Shape output_shape3 = output2.get_shape();
std::cout << "The shape of output kpts0:" << output_shape << std::endl;
std::cout << "The shape of output kpts1:" << output_shape1 << std::endl;
std::cout << "The shape of output matches0:" << output_shape2 << std::endl;
std::cout << "The shape of output matches1:" << output_shape2 << std::endl;

}

Feature Use Case

Application: Lightglue model for image registration, the input is a fixed-size image, the output is a dynamic shape. Openvino has a memory leak, I hope to solve this problem

Issue submission checklist

• The feature request or improvement must be related to OpenVINO

[barnasm1]

@ZzhangYyan Thank you for reporting this issue.
To help us investigate further, could you please provide more details on the method and tool you used for memory leak detection?
Additionally, it would be very helpful if you could share a complete, ready to use code snippet that reproduces the issue. This will help avoid any misunderstandings and ensure that our setups are identical.

[ZzhangYyan]

The code submitted above is the core reasoning part. The input is very simple, just two images. No operations are added to read the model. The normal reading method is sufficient. Test method: traverse the model inference data. The input image must be different for each inference. If the input image is the same each time, there will be no memory leak problem.

[barnasm1]

Thank you for the details provided. However, to effectively reproduce and diagnose the issue, we need a bit more information. The code snippet you included is helpful, but we require additional context to fully understand the environment and conditions under which the issue occurs.
To help us reproduce and resolve it, could you please provide the following details:

• Which OS are you using?
• Which compiler are you using? Please include the version.
• What compiling flags were used during the build process?
• Could you provide a compliable code snippet that demonstrates the issue?
• Which tool are you using to detect memory leaks? Please include the version and exact command did you use for memory leak detection?
• Which model from the list https://github.com/fabio-sim/LightGlue-ONNX/releases/tag/v1.0.0 do you use?

Providing these details will greatly assist us in reproducing and diagnosing the issue.

[ZzhangYyan]

Which OS are you using?
windows

Which compiler are you using? Please include the version.
visual studio 2022 v143

What compiling flags were used during the build process?

Could you provide a compliable code snippet that demonstrates the issue?
Test code: loop reasoning, registering two images, each loop input image needs to be different

Which tool are you using to detect memory leaks? Please include the version and exact command did you use for memory leak detection?
visual studio Snapshot
Take a snapshot after each inference to observe the difference in memory size

Which model from the list https://github.com/fabio-sim/LightGlue-ONNX/releases/tag/v1.0.0 do you use?
superpoint_lightglue_end2end_fused.onnx ,Use an end-to-end model.

[barnasm1]

~Take a snapshot after each inference to observe the difference in memory size

Please note that memory consumption snapshots might differ for the same inference with two different inputs. Comparing only memory consumption is not sufficient to confirm a memory leak. It is necessary to check the amount of memory allocation and deallocation.