rmdec: when using INT4 deinterleaving, error out if sub_packet_h <= 1.

Message ID 20120222172305.29B945E003@aruru.libav.org
State New
Headers show

Commit Message

Janne Grunau Feb. 22, 2012, 5:23 p.m.
Module: libav
Branch: master
Commit: e30b3e59a4f3004337cb1623b2aac988ce52b93f

Author:    Ronald S. Bultje <rsbultje@gmail.com>
Committer: Ronald S. Bultje <rsbultje@gmail.com>
Date:      Tue Feb 21 10:36:27 2012 -0800

rmdec: when using INT4 deinterleaving, error out if sub_packet_h <= 1.

We read sub_packet_h / 2 packets per line of data (during deinterleaving),
which equals zero if sub_packet_h <= 1, thus causing us to not read any
data, leading to an infinite loop.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org

---

 libavformat/rmdec.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

Patch

diff --git a/libavformat/rmdec.c b/libavformat/rmdec.c
index ee8abdd..ed16b07 100644
--- a/libavformat/rmdec.c
+++ b/libavformat/rmdec.c
@@ -265,6 +265,7 @@  static int rm_read_audio_stream_info(AVFormatContext *s, AVIOContext *pb,
         switch (ast->deint_id) {
         case DEINT_ID_INT4:
             if (ast->coded_framesize > ast->audio_framesize ||
+                sub_packet_h <= 1 ||
                 ast->coded_framesize * sub_packet_h > (2 + (sub_packet_h & 1)) * ast->audio_framesize)
                 return AVERROR_INVALIDDATA;
             break;