h264: stricter reference limit enforcement.

Message ID 20120314210239.419705E088@aruru.libav.org
State New
Headers show

Commit Message

Janne Grunau March 14, 2012, 9:02 p.m.
Module: libav
Branch: release/0.8
Commit: 4d343a6f47931a43fe8e4c9288a5068c76b843e0

Author:    Ronald S. Bultje <rsbultje@gmail.com>
Committer: Reinhard Tartler <siretart@tauware.de>
Date:      Tue Mar 13 16:26:44 2012 -0700

h264: stricter reference limit enforcement.

Progressive images can have only 16 references, error out if there are
more, since the data is almost certainly corrupt, and the invalid value
will lead to random crashes or invalid writes later on.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit e0febda22d0e0fab094a9c886b0e0f0f662df1ef)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>

---

 libavcodec/h264.c |   13 ++++++++-----
 1 files changed, 8 insertions(+), 5 deletions(-)

Patch

diff --git a/libavcodec/h264.c b/libavcodec/h264.c
index 95a672b..e0eb8e1 100644
--- a/libavcodec/h264.c
+++ b/libavcodec/h264.c
@@ -3020,6 +3020,8 @@  static int decode_slice_header(H264Context *h, H264Context *h0){
     h->ref_count[1]= h->pps.ref_count[1];
 
     if(h->slice_type_nos != AV_PICTURE_TYPE_I){
+        int max_refs = s->picture_structure == PICT_FRAME ? 16 : 32;
+
         if(h->slice_type_nos == AV_PICTURE_TYPE_B){
             h->direct_spatial_mv_pred= get_bits1(&s->gb);
         }
@@ -3029,13 +3031,14 @@  static int decode_slice_header(H264Context *h, H264Context *h0){
             h->ref_count[0]= get_ue_golomb(&s->gb) + 1;
             if(h->slice_type_nos==AV_PICTURE_TYPE_B)
                 h->ref_count[1]= get_ue_golomb(&s->gb) + 1;
+        }
 
-            if(h->ref_count[0]-1 > 32-1 || h->ref_count[1]-1 > 32-1){
-                av_log(h->s.avctx, AV_LOG_ERROR, "reference overflow\n");
-                h->ref_count[0]= h->ref_count[1]= 1;
-                return -1;
-            }
+        if (h->ref_count[0] > max_refs || h->ref_count[1] > max_refs) {
+            av_log(h->s.avctx, AV_LOG_ERROR, "reference overflow\n");
+            h->ref_count[0] = h->ref_count[1] = 1;
+            return AVERROR_INVALIDDATA;
         }
+
         if(h->slice_type_nos == AV_PICTURE_TYPE_B)
             h->list_count= 2;
         else